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ABSTRACT 


The development of the Logistics Management Decision Support System 
(LMDSS) by the Naval Air Systems Command (NAVAIR) has been on going 
since 1991. Originally conceived as a strategic Decision Support System (DSS), 
LMDSS has instead evolved into a Web portal for data analysts to access 
NAVAIR's Aviation Maintenance and Material Management (AV-3M) data. 
LMDSS is more accurately described as a Web-based, Management Information 
System (MIS) than as a DSS. 

This research examines the information needs and decision process 
requirements of LMDSS users. Focus groups, interviews, and a Web-based 
survey were conducted to collect decision support and data requirements from 
Fleet customers. User perceptions, feedback, and recommendations to improve 
LMDSS are described and analyzed. Historical insights into the development 
history of LMDSS are introduced as a lessons learned for future NAVAIR 
software teams. 

Problems identified by users are presented followed by specific — 
recommendations for solutions. A protype model developed for this thesis is 
found in the appendices as an example of how a modeling capability could 
enhance the ability of LMDSS to better support strategic, unstructured decisions.. 
Recommendations are provided for future research. 
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L INTRODUCTION 


A. BACKGROUND 


The first effort toward building a Decision Support System (DSS) for program 
managers at the Naval Air Systems Command (NAVAIR) began in 1991. NAVAIR's 
Program Managers, Air (PMAs) and Assistant Program Managers, Logistics (APMLs) 
needed a tool to help them investigate alternatives and make optimal, unstructured 
decisions in their efforts to reduce life cycle program costs while maintaining readiness. 
In 1994, such a tool was completed and named the Logistics Management Decision 
Support System (LMDSS). 

When Rear Admiral Donald Eaton (NAVAIR 4.0) and others were briefed on the 
LMDSS program in 1994, many were excited by its potential. One APML even reported 
attaining a $42 million cost avoidance when LMDSS was used to identify and solve a 
problem in the F-14 avionics system. It appeared that LMDSS might just be the tool 
PMAs needed to better manage the life cycle support costs of their programs as harsh 
budget cuts loomed on the horizon. | 

Years later, as the Logistics Program Department Head at the Naval Postgraduate 
School, Rear Admiral (retired) Donald Eaton wondered whatever became of the LMDSS 
program that showed so much promise back in 1994. One of the goals of this thesis is to 
find out what happened to the LMDSS program. Does LMDSS currently meet the 


requirements of its users? 


B. OBJECTIVES 


The primary objective of this research is to investigate the information needs and 
, decision process requirements of current LMDSS users. A secondary objective is to 
document how and why LMDSS took eight years to transition from a strategic level DSS 
to a Web based Relational Database Management System (RDBMS) interface for Fleet 








data analysts at all levels. A final goal is to provide our sponsor, NAVAIR 3.6.2, with 


user feedback and research results to assist in the management of the LMDSS program. 


C. RESEARCH QUESTIONS 


The primary research question is: "Does LMDSS meet user requirements?" Other 
major research questions are: 

e Howcan LMDSS be improved to better meet managerial needs? 

e What are the significant decisions users would like LMDSS to support? 

e How has LMDSS been funded? 

e Howcan NAVAIR improve the LMDSS program? 

e What can future software developers learn from studying the LMDSS project? 

e What are the key attitudes and opinions of LMDSS users? 


D. ORGANIZATION OF THE THESIS 


This study documents some of the key events that occurred during the eight-year 
effort to develop LMDSS and examines whether or not LMDSS currently meets the 
requirements of its users. Chapter II is a review of previous LMDSS research, DSS 
literature, and results from current software development research. Chapter II] reviews 
the history of the LMDSS project and discusses how it has been funded. Chapter IV 
documents a detailed description of the research methodology used in this thesis. Chapter 
V reports the research findings. Chapter VI presents an analysis of the findings and 
provides answers to the above research questions. Chapter VII provides a continued 
discussion of what the research results mean with conclusions and corresponding 
recommendations. A prototype model is presented in the appendix section as an example 
to the reader of how modeling can assist managers in making strategic, unstructured 


decisions. 








ii. LITERATURE REVIEW 


A. PREVIOUS LMDSS RESEARCH 
1. LMDSS Overview 


The LMDSS concept was first introduced by the Naval Air Systems Command 
(NAVAIR) and the Naval Inventory Control Point (N AVICP), formally called the 
Aviation Supply Office (ASO), in 1991. The concept involved the development of a 
decision support system (DSS) to assist NAVAIR logistics and weapon system managers 
in reducing the life cycle support costs of aviation systems while maintaining readiness. 
Although the LMDSS application was completed, delivered, and operational for a short 
period in 1994, it was placed back into development to be restructured as a World Wide 
Web (WWW) based tool. As of the summer of 1999, LMDSS is still considered a 
prototype rather than a fully functional software application. 

The only published research that specifically addresses the LMDSS application 
was completed by Moore and Snyder (1998) at the Naval Postgraduate School. They 
assessed the capability of LMDSS to meet the information requirements of NAVAIR's 
logistics managers. They also gauged whether the LMDSS prototype's capabilities 
satisfied the criteria of a DSS. 

LMDSS is unusual because of the time it has been under development (over 8 
years). As a result, we chose to expand the focus of research to include software 
development and acquisition issues in addition to user requirements. The goal is to help 
explain how LMDSS grew from a three-year project to a seemingly endless software 
development effort. 

2. LMDSS Research Findings 


Moore and Snyder (1998) concluded that: 


e Because of its lack of modeling and sensitivity analysis, LMDSS is not a 
DSS. It is actually a relational database management system (RDBMS) 
interface that improves data accessibility. 








° APMLs are only one of many potential user groups of LMDSS. 


6 LMDSS meets the requirements of surveyed logistics managers to support 
Affordable Readiness initiatives. 


® Lack of modeling, graphics, and sensitivity analysis capabilities limits the 
identification, analysis, and comparison of Affordable Readiness 
initiatives using LMDSS. 

@ Data quality is both a real and perceived problem. 

© The environment in which aviation program management decisions are 
made limits the effectiveness of LMDSS in measurably reducing life-cycle 
costs. 


a) Research Weaknesses 


Moore and Snyder's (1998) research targeted NAVAIR's APMLs as the 
primary users of LMDSS. Survey results from the nine APMLs contacted revealed only 
one respondent had personally used LMDSS. Their interviews revealed each of the 
APML program offices had its own unique strategy for accomplishing logistics and 
maintenance data analysis and collection. Some offices hired civilian contractors while 
others used various mixtures of in-house personnel and personnel assigned to lower 
echelon commands. They discovered data analysts rather than APMLs and PMAs were 
the actual users of LMDSS. Although LMDSS was originally designed to support PMAs 
and APMLs, its use has been delegated to lower level analysts. As a result, the LMDSS 
prototype has evolved from a PMA/APML decision tool to a RDBMS interface for Fleet 
data analysts. The requirements identified by the nine APMLs in Moore and Snyder's 
thesis are not representative of the needs of the larger, more varied population of over 
800 LMDSS users. 

The fact that LMDSS is still in the prototype phase and thus not widely 
' used was mentioned as a significant research constraint (Moore and Snyder, 1998). 


Although LMDSS was scheduled to be operational by the summer of 1998, it was still 


undergoing quality assurance testing as of the summer of 1999. The constraints posed 











when querying users about a prototype software tool that was not fully operational 


continue to be a problem for researchers. 


b) Research Strengths 


Moore and Snyder's (1998) thesis work reiterated how important it is for 
NAVAIR's PMAs to make smart decisions concerning life cycle support costs. The data 
collected showed that PMAs require more than a measuring tool to display costs. To 
make optimal decisions, PMAs and APMLs need the ability to test alternative courses of 
action to see how controllable and uncontrollable variables affect life-cycle costs and 
readiness. According to the APMLs, a fully functional DSS 1s still needed at the PMA 
and APML levels to make such decisions. 

Previous research verified that LMDSS could not be classified as a DSS 
due to its inability to support modeling, graphics, and sensitivity analysis. Such 
limitations significantly degrade LMDSS's usefulness as an effective decision support 
tool for the PMAs and APMLs. One APML that Moore and Snyder interviewed stated, 


LMDSS is good at "big picture” and indicating "where to go look" 
but falls short of communicating details with ease or indicating "why" a 
system measurement is as it is..."LMDSS can help answer the what, but it 
can't help me with the why or the how." (Moore and Snyder, 1998, p. 86) 


Moore and Snyder (1998) identified problem areas in the LMDSS 
environment that significantly degraded its effectiveness. Data quality issues, difficulties 
with depot cost data, and the need for users to understand the origin and derivation of 
data to fully utilize and trust LMDSS were noteworthy findings. It was noted that the 
environment of a typical PMA did not encourage or reward risk taking and creativity 


when considering life cycle costs. They wrote, 


The current environment encourages short-term decisions that 
compromise life-cycle decisions. The LMDSS can help identify and 
justify decisions to reduce life-cycle costs, but other factors are driving up 
these same costs. (Moore and Snyder, 1998, p. 93) 








Instances where NAVAIR failed to communicate adequately with LMDSS 
users were also mentioned as potential problems. An example was the LMDSS web site's 
failure to clearly identify itself as a prototype site with only a partial database. This 
caused numerous misunderstandings at the user level. Such communication errors 
directly affected user perceptions about LMDSS's relevancy and led to a lack of 
acceptance and trust. (Moore and Snyder, 1998) 

3. Identification of LMDSS Users 


Moore and Snyder (1998) identified only one APML who was an actual LMDSS 
user. Most of the real users were found to be military and civilian data analysts at all 
levels within the Fleet. Much of the earlier DSS literature suggested the use of DSS 
might be more appropriate at only the higher levels of management. This was the view of 
those who originally developed LMDSS back in 1991. More recent DSS literature now 
claims the use of a DSS is appropriate at any management level. (Ariav and Ginsberg, 
1985) 

While much of the literature emphasizes DSS use by top management, in reality, 
middle management and professional staffers are often the real hands-on users of a DSS 
(Sprague, 1996). 

This does not mean that DSS is not used by top management. Rather, 

what often happens is that senior executives request information from a 


staff assistant who uses a DSS to obtain information. (Sprague, 1996, p. 
103) 


This finding appears to confirm what Moore and Snyder uncovered in their thesis. 
The PMAs and APMLs contacted by them admitted delegating the use of the LMDSS 
application to lower level analysts. This delegation provides some insight into how 
LMDSS transitioned from a strategic level decision tool to an analysis tool for Fleet data 
analysts. 

One study conducted by Snead and Harrell discussed the gap between the 
capabilities of sophisticated computer-based information systems (CBIS) and the extent 





they are used by managers. Their conclusions emphasized the important role a manager's 
expectations play in whether or not he accepts and uses a CBIS. A strategy that includes 
more extensive manager involvement and training to support the use of a CBIS was 
strongly recommended. The study suggests more aggressive training and close user 
involvement may be the key to getting NAVAIR's logistics and maintenance managers to 
accept and use LMDSS when it becomes fully operational. (Snead and Harrell, 1994) 

A typical DSS user has much more latitude to ignore or circumvent the system 
than users of other more traditional CBIS. Research emphasizes how important it is for a 
DSS to earn the allegiance of its users by being valuable and convenient. (Sprague, 1980) 
Little (1970) called for DSSs to be simple, robust, easy to control and adaptive in order 


gain management support. 


B. DECISION SUPPORT SYSTEM LITERATURE 


On NAVAIR's web site, the LMDSS application is presented as a DSS which 
integrates data from maintenance, flight, cost, and material data bases into a structured 
and repeatable decision making process. In numerous NAVAIR briefs and plans, 
LMDSS is shown as Naval Aviation Logistics Data and Analysis (NALDA) II's primary 
analysis tool for historical Aviation Maintenance and Material Management (AV3M) 
data within NAVAIR's Integrated Weapons Systems Database (IWSDB). (NAVAIR, 
1999) 

To avoid misunderstandings when discussing a DSS, it is important to review the 
‘basic DSS definitions and terms provided by the literature. 

1. What is a DSS? 


An accepted definition of a DSS is one provided by Bhargava, Krishman, and 
Whinston (1994, p. 2): 





Traditional decision support systems (DSS) support individuals 
making semi-structured decisions in which mathematical (or other formal) 
models are used for the structured parts, leaving the decision maker to 
exercise judgement in handling the unstructured parts. Focusing on the 
choice-related tasks, DSS facilitate the use of formal modeling techniques 
in making complex decisions. Among the benefits claimed for these 
systems is that they facilitate the investigation of more alternatives, and 
support ad hoc query and analysis. 


Another definition defines a DSS as a computer-based information system, which 
provides interactive information support to managers making semi-structured and 
unstructured decisions. A DSS supports managers by using, (1) analytical models, (2) 
specialized databases, (3) a decision-maker's own insights and judgements, and (4) an 
interactive, computer-based modeling process. (O'Brian, 1993) 

Some terms in the above DSS definitions need to be clarified. An ad hoc query is 
a unique, situation-specific, unscheduled request for information. A structured decision 
occurs in situations where the procedures to follow in making a decision can be predicted 
and specified in advance. Often, the outcome of structured decisions can be predicted 
with relative certainty. An unstructured decision involves situations where it is not 
possible to know in advance what procedures to follow. Unstructured decisions involve 
too many random events, unknown variables, and hidden relationships to be able to 
follow any preset rules or checklists. A semi-structured decision is one where some of 
the decision procedures can be pre-specified, but not to an extent where it can lead to a 
definite decision. (O'Brian, 1993) | 

Currently the LMDSS application can support structured decisions and ad hoc 
queries, but is unable to assist with unstructured or semi-structured decisions due to the 
absence of modeling and sensitivity analysis. 


2. What is the Decision-Making Process? 


A classic model of a manager's decision process is depicted by Herbert Simon, a 


leader in the field of decision theory, who presented decision-making as consisting of 











three phases: intelligence, design, and choice. The intelligence phase involves detecting 
problems, collecting problem data, and analysis of the environment. The design phase 
includes setting criteria and objectives, creating alternatives, and evaluating the outcome. 
The choice phase consists of determining the outcome of selected alternatives and 
selecting the desired outcome consistent with the decision objectives. Simon's work is 
important because it built the theoretical foundation still used by software engineers to 
build DSSs. (Simon, 1977) 

Steven Alter added a-fourth phase to Simon's model calling it the implementation 
phase. This phase involved the process of putting a manager's decision into effect. It 
included building a consensus to support the decision, explaining the decision to the right 
people, and fostering the commitment to follow through with the decision. Alter's 
research into this implementation phase revealed a DSS's greatest impact was not in 
aiding the decision-making process but with explaining and justifying decisions already 
made. (Alter, 1980) 

3. Why do PMAs and APMLs need a DSS? 


Most researchers agree that managers are not completely rational decision- 
makers. An ideal rational decision-maker performs all three phases of Simon's model 
flawlessly. Such a decision-maker ensures all information is gathered and interpreted in 
an unbiased manner. He identifies all practical alternatives and evaluates them using 
unambiguous criteria. He then selects the best alternative based on a consistent and 
explicit set of weightings and trade-offs. (Alter, 1992) 

In the real world, however, managers don't have time to collect all decision- 
related information and fully explore all feasible alternatives. Clear, focused objectives 
and criteria are often not completely defined with more weight placed on intangible 
information. Alter (1992) identified eight common managerial decision-making flaws: 


e Poor Framing: Allowing a decision to be influenced excessively by the 
language used in describing the decision 


® Recency Effects: Giving undue weight to the most recent information 








® Primacy Effects: Giving undue weight to the first information received 


e Poor Probability Estimation: Overestimating the probability of familiar or 
dramatic events; underestimating the probably of negative events 

@ Overconfidence: Believing too strongly in one's own knowledge 

° Escalation Phenomena: Unwillingness to abandon courses of action that 


have been decided upon previously 


e Association Bias: Reusing strategies that were successful in the past, 
regardless of whether they fit the current situation 


® Groupthink: Overemphasizing group consensus and cohesiveness instead 

of bringing out unpopular ideas 

Simon developed the idea that managers "satisfice" by choosing a satisfactory 
course of action that is "good enough". His research found human decision makers have 
limited information processing capabilities and resort to satisficing when faced with time 
constraints, minimal information, and a limited ability to process all relevant 
information. (Simon, 1977) 

The purpose of a DSS is to convert decision making from an art into a scientific, 
more rational approach. Relying on a DSS can help managers avoid some of the common 
mistakes listed above. A DSS can be also be an invaluable tool to justify NAVAIR 
programs and decisions made by PMAs and APMLs. A DSS can help a manager frame a 
problem, collect more pertinent information, develop and evaluate feasible courses of 
action, test assumptions, and select the best solution to achieve the desired objectives. 
- The original vision of the LMDSS application was to provide such support to PMAs and 
APMLs in order to achieve life cycle cost savings and improve readiness in an 
environment of shrinking budget authority. 

The literature concerning DSS architecture contains various structural views of a 
DSS concept with each author using unique nomenclature to describe the parts. Most 
descriptions break down a DSS into three conceptual components: (1) a user/system 


interface, (2) data management, (5) and model management (Ariav and Ginzberg, 1985). 
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One view of a DSS describes it using a Dialog Data Model (DDM) paradigm. 
This paradigm states all DSSs contain, in one form or another, three major components: a 
Dialog between the user and the system, the Data supporting the DSS, and Models that 
provide the analysis (Sprague, 1996). Our search of the DSS literature was unable to 
uncover any author who did not list a model base or model capability as a key component 
of a DSS. 

4, Modeling within Decision Support Systems 


a) Capabilities of a DSS Model 


Models provide powerful analytical capabilities toa DSS. The capability 
of a DSS to simulate a process with a model and conduct what-if analysis is what 
- separates it from a traditional CBIS. Sprague (1980), Ariav and Ginzberg (1985) 
determined the key capabilities of the model component within.a DSS to include: 


e The ability to easily create new models and update parameters 


® Support of a model directory containing all available models 
@ The capability to interface with databases to retrieve data, store model 


output, and input data to other models through databases 
e Access to a library of model "building blocks" 


e Support of a Model Base Management System (MBMS) containing 
mechanisms to store, catalog, link and access models 


b) Four Types of Modeling Activities 


The four types of analytical modeling that can be found in a DSS are: (1) 
what-if analysis (2) sensitivity analysis (3) goal seeking analysis and (4) optimization 
analysis (O'Brian, 1993). . 

What-if analysis involves making changes to process variables or variable 
relationships and observing the resulting changes in the output variables. Sensitivity 


analysis is a special case where only one variable is changed repeatedly to see the affect 
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on other system variables. Goal seeking analysis sets a target value or goal and changes 
various variables until the goal is achieved. Optimization analysis is finding the optimum 


value for selected variables, given certain constraints. (O'Brian, 1993) 


Cc) Why Models are Hard to Implement 


The difficulties inherent in building models today are the very same ones 
Little (1970) wrote of in his often cited research. His conclusions may help to explain 
why LMDSS does not currently contain models. He noted four reasons why managers do 
not use models more widely: 


e Good models are hard to find. Models, which accurately depict 
management control and environmental variables, are difficult to build. 


e Good parameterization is even harder. Metrics and accurate, timely data 
are a mandatory requirement. Complex, high quality design work is 
required and expensive. 


e Managers don't understand the models. A manager is responsible for his 
decisions and will not trust the output from a model he does not fully 
comprehend. Often, a manager will choose a simpler analysis that he can 
grasp over a model whose assumptions are partially hidden and contain 
complex statistical manipulations. 


® Most models are incomplete. An incomplete model can pose a serious risk 
to optimization analysis. 


C. SOFTWARE DEVELOPMENT LITERATURE 


Since 1991, the LMDSS development effort has faced more than its share of 
obstacles. LMDSS meets the Standish Group's criteria for a challenged software project 
because it is over budget, has exceeded its original time estimate, and contains fewer 
features than originally specified (Standish Group, 1999). Research is now available 
which can help managers, developers, and clients better understand the risks and classic 
mistakes of the software development process. Research literature was selected to show 
how prevalent the LMDSS situation is among other organizations and to provide needed 


guidance to software developers seeking to bring such projects under control. Proven 
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fundamental strategies are presented which can significantly assist software development 
teams in their efforts to rapidly achieve their objectives within realistic cost constraints. 


1. The Software Crisis 


The Standish Group's (1999) research has shown only nine percent of software 
development projects within larger companies are completed on time and on budget. The 
average cost overrun for these projects was 178 percent. In 1995, American companies 
spent an additional $59 billion to complete software projects that exceeded their original 
time estimates. The average project overrun is 222 percent of the original time estimate. 
Excessive schedule pressure occurs in 75 percent of all large software projects and close 
to 100 percent of all very large projects (Jones, 1994). The average software developer in 


the United States works 48 to 50 hours a week (Krantz, 1995). 


In this environment, it isn't surprising that general job satisfaction of 
software developers has dropped significantly in the last 15 years 
(Zawacki 1993), and at a time when the industry desperately needs to be 
recruiting additional programmers to ease the schedule pressure, 
developers are spreading the word to their younger sisters, brothers, and 
children that our field is no fun anymore (McConnell, 1996, pg. xviii). 

Survey results show there are plenty of organizations struggling to get software 
development projects under control. Most are trying to overcome the same people, 
process, and technology issues faced by NAVAIR's LMDSS development team. The first 
step to improving the software development process is to identify the ineffective 
practices and classic mistakes. The second step involves a return to effective, proven 
fundamental software development practices. The third step is to manage identified risks 
to avoid major setbacks. (McConnell, 1996) 


2. Classic Mistakes Leading to the Ten Most Serious Risks to Software 
Development 


McConnell (1996) identified 36 classic mistakes in the software development 
process, which can lead to a failed program. These classic mistakes are described in 


Appendix A. Based on research results involving Software Productivity Research (SPR) 
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assessments, Capers Jones (1994) reported the mistakes that contribute the most to the 


failure of software projects as: 


Lack of historical software-measurement data 

Rejection of accurate measurement estimates 

Failure to use automated estimating and planning tools 

Excessive, irrational schedule pressure 

Creeping (new and unanticipated) user requirements 

Failure to monitor progress and to perform formal risk management 


Failure to use design reviews and code inspections 


From a total sample size of 365 respondents representing 8,380 software 


applications, The Standish Group (1999, p. 5) conducted four focus groups and 


interviews with various IT managers. Based on the previously discussed definition of a 


challenged software project, they identified ten factors, that contribute to such a project: 


Lack of user input 

Incomplete requirements and specifications 
Changing requirements and specifications 
Lack of executive support 

Technology incompetence 

Lack of resources 

Unrealistic expectations 

Unclear. objectives 

Unrealistic time frames 


New technology 
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Once the mistakes and ineffective practices have been identified, they must be 
corrected and replaced with proven management, technical and quality assurance (QA) 
fundamental practices. After completing a study of ten "best projects," one researcher 


concluded, 


If there was one high-level finding that stood out, it is that best 
projects get to be best based on fundamentals. All of us know the 
fundamentals for good software--the difference is that most projects don’t 
do them nearly so well and then get into trouble. (Hetzel, 1993, p. 151) 


3. Management Fundamentals 


Management fundamentals involve estimating the size and characteristics of the 
project (including software functionality and complexity), assigning adequate resources, 
creating a plan, and monitoring and directing the plan to stay on course. Effective 
estimation strategies are essential to developing a realistic plan upon which both 
managers and developers can depend. (McConnell, 1996) The best projects are 
characterized by strong up-front planning to define tasks and schedules (Hetzel, 1993). 


A software development project should include: 
e Estimation and scheduling 


e Determining how many people to have on the project team, what technical 
skills are needed, when to add people, who the people will be 


® Deciding how to organize the team 
e Choosing which lifecycle model to use 
@ Managing risks 


° Making strategic decisions such as how to control the product's feature set 

and whether to buy or build pieces of the product (McConnell, 1996, p.56) 

Once the plan is developed, it has to be monitored to ensure all cost, schedule, 

and quality milestones are met. Fundamental management tracking activities include: 


task lists, status meetings, status reports, milestone reviews, budget reports, and 
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management by walking around. Fundamental technical tracking events include 
technical audits, technical reviews, peer reviews, and quality gates to ensure functional 
objectives are met at the appropriate milestone. (McConnell, 1996) Not implementing 
tracking procedures properly can have disastrous effects. Jones (1995, p. 87) reported 
that, "Software progress monitoring is so poor that several well-known software disasters 


were not anticipated until the very day of expected deployment." 


If you don't track a project, you can't manage it. You have no way of 
knowing whether your plans are being carried out and no way of knowing 
what you should do next. You have no way of monitoring risks to your 
project. Effective tracking enables you to detect schedule problems early, 
while there's still time to do something about them. (McConnell, 1996, p. 
57-58) 


Collecting metrics is a critical management requirement to track and analyze 
software quality and productivity. Most projects collect measurements of cost and 
schedule targets, but such data doesn’t provide the insight needed to reduce program 
costs or shorten time schedules. A knowledge of specific metrics needed to analyze 
project status, quality, and productivity is required to know whether or not a project's 
development speed is improving or degrading. (McConnell, 1996) | 


The importance of metrics in planning and estimating is reinforced by Jones 


(1996, p. 104): 
Software failures are caused primarily by errors and poor judgement 
on the part of managers, executives, and clients - not errors made by the 
technical teams. The root cause of these failures is the lack of accurate 


measurement data, which blinds management and clients to what is 
possible and what might be possible. 


4. Technical Fundamentals 


Requirements management involves gathering and documenting user 
requirements, tracking the project design and code, and managing all changes for the 


entire project (McConnell, 1996). A Standish Group (1994) survey of over 8000 
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software projects found lack of user input, incomplete requirements, and changing 
requirements were the top three reasons for late, over budget projects with less than 
anticipated functionality. The fundamentals of requirements management are: 


e Requirements analysis methodologies including structured analysis, data 
structured analysis, and object oriented analysis 


e System modeling practices such as class diagrams, dataflow diagrams, 
entity-relationship diagrams, data dictionary notation, and state transition 
diagrams 

e Communication practices such as Joint Application Development, user- 


interface prototyping, and general interview practices 


@ The relationships between requirements management and the different 
lifecycle models including evolutionary prototyping. (McConnell, 1996, p. 
62) 


Research has shown that the cost of reworking software is smaller by a factor of 
50 to 200 in the earlier phases of the development cycle than waiting until the 
construction or maintenance phases to correct them. This puts a high emphasis on early 
error detection and getting requirements correct the first time. (Boehm and Papaccio, 
1988) 

Software configuration management is defined as the “methodical storage and 
recording of all software components and deliverables as projects are under 
development” (Jones, 1994, p. 556). It includes evaluating proposed software changes, 
monitoring changes, updating planning documents, keeping records of all previous 
versions, and tracing backwards to original requirements. Automated tools are available 
which can help developers synchronize, cross-reference, integrate, and update 
specifications, graphics, user information, source code, defect repairs, and test cases. 
Configuration management is a requirement for military software and often represents a 


significant cost factor. (Jones, 1994) 
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Poor configuration management can allow developers to write code that is 
incompatible with code written by other team members resulting in expensive rework. 
Jones (1994) reported that 30 percent of U.S. software producers had no configuration 
control automation of any kind. About 30 percent had some form of source code 
automation, but no automation for documentation, test cases, and defect tracking. For a 
nominal 10,000 function point project, Jones reported an added cost of 168 man-months 
for projects that failed to use automation. For large projects, he reported configuration 
control to be a critical path item. 


=i Quality Assurance Fundamentals . 


a) The Cost of Software Defects 


Research has overwhelming shown that identifying and correcting 
software defects early in the development process can generate significant savings in 
both time and money. Every hour spent on defect prevention will reduce a project's 
rework time by three to ten hours. After surveying approximately 4000 projects, Jones 
reported one of the most common causes of schedule overrun was poor quality software. 
(Jones, 1994) Another survey conducted at 59 government and industry software sites 
examined 296 software projects and found that. quality assurance was a significant 
problem at 63 percent of the sites assessed (Kitson and Masters, 1993). 

Late projects are particularly vulnerable to shortening the QA process in 
response to pressures to meet deadlines. Surveys have found that software products 
developed under excessive schedule pressure have reported up to 4 times the number of 


normal defects (Jones, 1994). 


b) Strategies in Achieving Quality Software 


The most common strategy for finding software defects is testing. Testing 
should include unit testing, where the developer checks his own code, and system testing, 


where an independent tester checks the application to verify it works as advertised. The 
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best way to implement testing is to conduct it in a manner where errors can be identified 
and corrected as early as possible. (McConnell, 1996) 

A key element of the QA process is formal technical reviews. Technical 
reviews include formal design reviews, walkthroughs, internal code reading, and 
inspections. 

Formal design reviews provide feedback to management concerning the 
status of the software project under development. They typically occur at the end of the 
various development phases and based on their results, management decides whether or 
not the product is ready to proceed to the next phase. 

Walkthroughs are meetings where two or more developers form a peer 
review team to inspect specific sections of a co-workers code with the purpose of 
improving quality. Walkthroughs are essential to the QA process because they can be 
used to detect software defects well before normal testing is accomplished. (McConnell, 
1996) Documenting these reviews 1s accomplished through a walkthrough summary 
report that contains what was reviewed, who reviewed it, and what findings and 
conclusions were reached. 

Code reviews are more formal than walkthroughs and usually limited to 
inspections of only code. A code review consists of a developer passing his work to two 
or more reviewers. The reviewers read the code and report any discovered defects to the 
author of the code. (McConnell, 1996) A study at NASA's Software Engineering 
Laboratory by Card (1987) found that internal code reading procedures detected twice as 
many defects per hour of effort as testing. 

Software inspections are technical reviews where each design component 
is inspected at the requirements, preliminary design, detailed design, and code phases. 
Inspections use trained peer teams of three to eight people with each person playing a 
specific role. A team leader hands out the work to be inspected before the formal 
meetings. Reviewers examine the work using checklists and arrive at the team meetings 


with their results. At team meetings defects are identified and a plan of action for 
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correcting them is developed. Throughout the inspection process data on all defects, 
hours spent correcting them, and hours devoted to inspections are collected to assist in 
the improvement of the software development process. (McConnell, 1996) A study by 
Russell (1991) found that for large programs, each hour spent on inspections avoided an 
average of thirty-three hours of maintenance. He also noted that inspections were up to 
twenty times more efficient than testing. 


6. Risk Management 


Many of the proven, fundamental software development practices described are 
intertwined and rely on each other to be successfully implemented. Jones (1996, p. 103) 
states, 

A well-formed risk analysis and milestone-tracking program for 


software projects depends, quite simply, on successful completion of 
formal design and code inspections. 


The purpose of a formal risk management program is to identify, address, and 
minimize sources of risk before they create significant obstacles to a software project. 


The elements of risk management include risk assessment and risk control (McConnell, 


1996). 


a) Risk Assessment 


Risk assessment is made up of risk identification, risk analysis, and risk 
prioritization. Risk identification simply identifies the factors that represent a risk to a 
software project's successful completion. The most common Management Information 


System (MIS) software risks identified in the literature are found below: 
® Creeping user requirements 
® Excessive schedule pressure 
e Low quality 


e Cost overruns 
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e Inadequate configuration control (Jones, 1994, p. 29) 


Risk analysis involves determining the impact of each risk identified. Risk 
analysis can involve estimating probabilities, the size of losses, and the levels of 
exposure in time periods such as weeks or months. Risk analysis may also involve the 
use of performance and cost models, decision analysis, and quality factor analysis. 

Risk prioritization follows the identification and analysis phases with the 
development of a pnirity list of risks so risk reduction efforts can be focused on the 


greatest threats. Boehm and Papaccio (1988, p. 1466) noted, 


80 percent of the rework costs typically result from 20 percent of the 
problems...The major implication of this distribution is that software 
verification and validation activities should focus on identifying and 
eliminating the specific high-risk problems to be encountered by a 
software project, rather than spreading their available early-problem- 
elimination effort uniformly across trivial and severe problems. 


b) Risk Control 


Once risks have been identified, measured, and prioritized, they have to be 
controlled and minimized. Risk control activities include risk management planning, risk 
resolution, and risk monitoring. 

The purpose of risk management planning is to develop a written plan to 
address the highest priority risks identified earlier. The plan can be as simple as a 
paragraph for each risk detailing the who, what, where, how, and why of each risk's 
management strategy. The plan should describe how risk management strategies will be 
monitored, how eliminated risks will be closed out, and what new risks have been 
identified. (McConnell, 1996) 

Risk resolution is the implementation of the msk management plan to 
avoid, reduce, transfer, eliminate or control identified high priority risks. Prototypes, 


simulations, and benchmarking may be used in this phase to investigate and resolve risks. 
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Risk monitoring also involves implementing the risk management plan by 
tracking risk management actions and documenting identified emerging risks. Risk 
monitoring procedures often include milestone tracking, risk reassessments, and 
documenting corrective action. 


I Research Recommendations for Avoiding Disaster 


Based on his years of researching hundreds of software projects, Jones (1996, p. 
104) supplements the proven practices previously described with some additional 


recommendations to achieving success in software development efforts: 


® Look at the actual results of similar projects 


° Make planning and estimating formal activities 

® Plan for and control creeping requirements 

® Use formal fispecuons as milestones for tracking project progress; and 

@ Collect accurate measurement data, during the current project, to use with 
future projects 
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fii BACKGROUND INFORMATION 


A. DEVELOPMENT HISTORY 
1. Background Information 


The end of the Cold War created a new world order with the United States as the 
only surviving superpower. After the fall of the Berlin Wall, the American public and 
members of Congress anticipated massive cuts in the military. An immediate sense of 
increased security prevailed, resulting in demands for heavy reductions in defense 
spending. The term “peace dividend” was coined and popularized as it was felt that the 
savings generated would be applied to domestic programs and to help lower the deficit. 
Over the next several years, the defense budget was drastically reduced, to the point 
where the fiscal year 1999 defense budget represented a 39 percent drop from spending 
levels in the 1980s. The defense budget is now equal to just 3.2 percent of our gross 
national product, a level not seen since the 1930s. : 

The magnitude of these reductions was felt throughout the Department of 
Defense. Military bases were closed, ships were decommissioned, entire air wings were 

disestablished and thousands of fully trained troops were discharged. Acquisition 
| programs that had been funded for years were severely restricted or canceled out-right. 
The only thing in the military that wasn’t cut back was the operational tempo. Fewer 
troops were expected to fulfill the same requirements with older and less reliable 
equipment. Operating and support costs skyrocketed as aging aircraft, tanks and ships 
were pushed beyond their expected useful life in order to meet operational needs. 

Program managers, trying to do more with less, were forced to look closely at 
their programs and examine all expenditures in an effort to eliminate unnecessary cost 
drivers. For most program managers, this was a tedious and time-consuming task; for 


some it was simply impossible. The sheer volume of data, which were distributed in 
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technically diverse and geographically separated databases, made the job prohibitively 
expensive. | 

Analysts at NAVAIR 4.0 and NAVICP, formerly the ASO, were charged with the 
task of assessing the logistics health of aviation programs, focusing primarily on 
readiness, supportability, and cost. This involved the time-consuming manual efforts of 
pouring though the available data in an effort to minimize costs. After several months of 
effort, they concluded that the information systems available to them were inadequate for 
the job. If they were to meet the demands of cost reductions through analysis of the 
disparate, poorly maintained databases, program managers sorely needed a tool that 
could provide a central data repository and support the analysis of the data. 

With this in mind, steps were taken within NAVAIR to assess the feasibility of 
designing and implementing a DSS specifically to support logistics decisions. In this 


manner the concept of LMDSS was founded. 


2. LMDSS Fundamental Structure 


As stated above, the driving requirement behind the push forthe development of a 
DSS for logistics management was the need for a tool to facilitate measurements to use 
in the planning and identification of targets for cost reduction. The original vision 
statement for LMDSS required the detailed assessment and development of a DSS which 
would be the primary tool for APML/Weapon System Managers (WSM) to achieve a 
“continuous, measurable reduction in life-cycle costs while maintaining operational 
readiness and sustainability.” (Naval Air Systems Command, 1993). 

The specific goals of the initial LMDSS program as designed within the Naval 
Aviation Logistics Data Analysis (NALDA) system were as follows: 


e To be the primary DSS to achieve cost-effective logistics management 
e Improve more timely (daily) receipt of AV3M and configuration data 


@ Create cost-effective consolidation of central, upline AV-3M data systems 
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e Provide the ability to access centralized Fleet-wide, near real time 
operational and readiness data from NALDA in accordance with DoD data 
security regulations. (Naval Air Systems Command, Program Manager's 
Charter, 1997) 


In order to meet these objectives, it was essential to integrate vast quantities of 
diverse data contained in multiple legacy stovepipe databases into one central repository. 
The LMDSS requirement document (LMDSS Requirements Document/Statement of 
Work, 1993) also identified other key features as being essential to a successful program. 
These included: 


e Timely, precise responses to queries independent of type data or 


type/model/series (TMS) 

6 A user friendly system that did not require extensive computer knowledge 
or training 

® Ease of access to the system as well as maximizing the eligible ae of 


personnel who could access the system 


e Detailed statistical packages embedded in the program to provide 
projections of outcome based upon changes to maintenance and supply 
parameters 

® Both a structured modular approach to data recovery and an ad hoc 


Structured Query Language (SQL) capability 
e Assist tools to facilitate easy development of queries 


® Integration of the depot level AV3M data and the Naval Depot Logistics 
Management System (NDLMS) into the NALDA phase II existing 
database structure 


@ Graphical User Interface (GUI) capabilities designed to produce — 
presentation quality graphics on data obtained from queries 


e Inclusion of an Evaluation and Selection of Alternatives Module (ESM) 
based upon return on investment principles 
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® Inclusion of implementation and status tracking modules to document the 
actual implementation of actions recommended in the ESM module and to 
establish an audit trail | 


According to the initial Statement of Work (SOW), rapid prototyping was the 
software development method used to develop the initial prototype. The SOW also 
required the LMDSS software development teams to maintain maximum flexibility and 
“react with rapid turn around to additional requirements and changes to requirements 
levied by the LMDSS functional design team” (LMDSS Requirements Document/SOW, 
1993). 

At this time, the process for Major Acquisition of Information Systems (MAIS) 
was still a relatively new concept. Exacting measures and legislation were not in place to 
help identify means of formally structuring software development processes. Most 
legislation was developed in response to unforeseen difficulties arising from haphazard 


and unstructured software development processes. 


B. LMDSS’ ROLE IN SUPPORTING NAVAL AVIATION FOR THE 21°? 
CENTURY 


iF LMDSS Within the NALDA System 


LMDSS is currently a sub-component of the NALDA Phase II project. NALDA 
has been operational since the early 1980s, although LMDSS was not formally initialized 
until the early 1990s. NALDA is the Navy and Marine Corps central aviation 
maintenance and logistics Automated Information System (AIS). Its mission is to 
provide Integrated Logistics Support (ILS), engineering functions and achieve readiness 
and affordability process improvements through the implementation of advanced data 
collection, compilation and management techniques (Naval Air Systems Command, 
| Mission Need Statement, 1997). Other applications that fall under the umbrella of the 


NALDA system include logistics planning, management, administrative, budgeting and 
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resource allocation for aviation weapon systems and related support equipment. For a 
more detailed description of NALDA I, refer to Appendix B. 

NALDA Phase I currently operates on an AMDAHL 5995 mainframe located at 
the Defense MegaCenter in Mechanicsburg, Pennsylvania, but it is in the process of 
being phased out. This system is redundant, consists of outdated stovepipe systems with 
restricted functionality, and does not meet the requirements as set forth in current 
doctrine. In addition, many of the older legacy systems are not Year 2000 (Y2K) 
compliant and the Defense Information Systems Agency (DISA) has mandated all such 
systems be made Y2K compliant or be shut down by 15 March 1999. The NALDA 
system is scheduled to be completely converted to an IBM RS/6000 environment using 
the Advanced Interactive eXecutive (AIX) operating system and employing an Oracle 7.3 
relational database management system by the end of second quarter FY99. The current 
mainframe system, a Reduced Instruction Set Computing (RISC) SP machine, is 
operational but multiple unforeseen difficulties with the data loads has caused delays in 
the schedule. Parallel operations were planned to be in effect for a three month period 
from January to March 1999 in order to minimize the risk of decreased performance 
while identifying glitches in the new system. During this time period, NALDA I is 
scheduled to run on the server at Mechanicsburg while NALDA II runs concurrently on 
the server at NAS Patuxent River. As of May 1999, the dual system approach was 
extended to continue through the end of the fiscal year. 


2. LMDSS Within the Department of Navy (DON) Chief Information 
Officer (CIO)/Global Combat Support Structure (GCSS) 


The Office of Secretary of Defense Certification (December 21, 1995) for the 
NALDA system defined the NALDA migration strategy. The NALDA II program, when 
completed, will ensure compliance with the Department of the Navy Information 
Technology Strategic Plan which requires logistic support applications provide a 
seamless blend of operational and administrative information to all users (Naval Air | 


Systems Command, Program Manager's Charter, 1997). In addition, the plan was 
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structured in accordance with the Technical Architecture Framework for Information 
Management because it established a lorig-range goal of open systems architecture. It 
also met Defense Information Infrastructure (DI) Navy Common Operating Environment 
(COE) criteria. NALDA Phase II plans meet the Level 5 DI[ COE standard as defined in 
the Integration and Runtime Specification which enables it to be interoperable with other 
Level 5 or higher DIT COE systems like the Joimt Maritime Command Information 
System (JMCIS) or the Global Combat Support System (GCCS). (Naval Air Systems 


Command, Operational Requirements Document, 1997) 


C. CURRENT LMDSS ENVIRONMENT 
i. Initial Prototype Development 


The LMDSS Statement of Work was developed in 1993. The initial prototype 
was completed in 1994. It used the current NALDA IBM RISC System/6000 Model 970 
at ASO Philadelphia and other regional machines located at NADEP and Naval Air War 
Center (NAWC). The relatively new X-windows specifications, as developed by the 
Massachusetts Institute of Technology, were incorporated as the standard for GUIs. X- 
client and X-server structures were used. All programming on the initial prototype was 
done in Ada programming language. AIX operating systems which are IBM’s version of 
the single user version of MULT “ICS” (UNIX), were used exclusively and Transmission 
Control Protocol/Internet protocol (TCP/IP) was the only protocol used in NAVAIR's 
wide area network (WAN) connectivity plans. In addition, the Oracle relational 
database concept was the basis for the organizational matrices of the LMDSS database. 

The prototype worked well, but several of the initial requirements were not met. 
The UNIX operating system was unfriendly to all but the most highly trained computer 
personnel. The X-client and X-server implementation mandated the usage of a direct 
‘ local area network for personal computer accessibility. For those personnel on a WAN, 
the X-windows required a S6KBit/sec connectivity rate. Although that rate is fairly 


common in 1999, five years ago it was difficult and expensive to obtain reliable T-1 line 
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capability. Only the PMAs and APMLs located at NAS Patuxent River could get access 
to LMDSS. The contractors and analysts hired by PMAs for the majority of their data 
analysis did not have access to LMDSS because they were not part of NAS Patuxent 
River WAN. At this point, LMDSS was still considered a logistics management tool for 
high level logisticians, and according to later surveys, most of them were simply not 
using it (Moore and Snyder, 1998). 


2. Transition to Web Based Format 


In 1995, NAVAIR determined the detriments of the existing prototype did not 
meet the requirements as set out in the SOW. The decision was made to abandon the 
UNIX/X-windows environment in order to develop a web-based system with Common 
Graphics Interface (CGI) scripting. The Ada programming language was abandoned for 
HyperText Markup Language (HTML) and Practical Extraction and Report Language 
(PERL) because it was not suited for a major database management system. Although 
these decisions directly impacted major software development issues, the user group was 
neither expanded nor queried for inputs. A thorough risk analysis to determine the impact 
of these decisions was also not conducted and documented. The project fell behind the 
scheduled baseline and costs exceeded earlier estimates. As shown in the review of the 
software development literature, this is a common occurrence within the software 
industry. The Standish Group (1999) reported only nine percent of software developed 
by larger companies come in on-time and on-budget. Jones (1994) reported that most 
software development projects are cancelled because they fail to properly conduct 
requirements analysis, identify user needs, and conduct risk analysis/reduction 
assessments. As a result, instead of anticipating problems, the LMDSS team fell into the 
practice of reacting to events as difficulties arose. 


3. Integration into NALDA Program 


It was shortly thereafter that the decision was made to make LMDSS the basic 
interface to all NALDA data. The original LMDSS contract had been awarded to a small 


software development firm. It quickly became apparent the firm did not have the 
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expertise in HTML, CGI Scripting, ActiveX and data warehousing to meet the needs of 
the revised program specifications. When their contract was up, a major programming 
contractor was selected to replace the small business. Several of the firm's former 
employees stayed on as Federal Government Service workers. During this transition the 
LMDSS programmer pool was reduced from over thirteen full time programmers, to only 
three who were concurrently seeking other employment. It is presumed that a large chunk 
of corporate knowledge was lost in this contractor change, as not all programmers were 


hired on by the new company. 


Component - Upline 





eel how! 
- qe 


Figure 1 NALDA Upline Application 
LMDSS was changed once again from the basic interface for all of NALDA, to its 
current status as just the component of the NALDA II system that provides access to the 
reliability and maintenance portion of the NALDA II database structure. Figure 1 
displays the current NALDA upline configuration and includes the new functional 
relationship of LMDSS within the NALDA II system (NAVAIR 3.6, 1999). LMDSS 
does not do configuration management (CM), although it helps track CM data. When it 
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first was produced in 1994, it was hoped by NAVAIR that LMDSS would be the end-all, 
solve-all for any questions concerning reliability, readiness, cost, and configuration 
management for the Navy's aircraft. This is no longer the case; however, when the 
development of LMDSS is complete, it will have all the detailed data from 
approximately 2.5 million maintenance action records/month. Currently, NAVAIR has 
archived 32 months of maintenance data online and 11 years of flight data (Interview 
with Mr. Joe Joseph, 1998). 

Unfortunately, under the new NALDA II Acquisition Program Baseline (APB), 
LMDSS developers were forced to meet the deadlines established for the Milestone I] 
Decision Authority. The first increment, which consisted of the consolidation of the 
configuration management and AV-3M systems, was to be completed by February 1998. 
In the process of transitioning from one software language and platform to another and to 
meet the goals of the NALDA program, many DSS items were discarded. Graphics 
capabilities and some of the statistical forecasting and modeling modules were scrapped. 

What makes the current application of LMDSS useful is it’s summary tables that 
can provide, for example, the top five items in the last five years with the worst reliability 
performances for the maritime patrol aircraft (P-3) community. Another strength is its 
ability to drill down from the summary data to specific part numbers and locations. Even 
with its reduced capabilities caused by the multiple changes in software format and 
program structure, the remaining capabilities will enable NAVAIR program managers to 
identify areas to save money so they can use the funds to purchase more important items 
that are not funded. In short, doing more with less. 


4. Current Program Funding 


Funding reductions have had the most detrimental impact on the LMDSS 
program. LMDSS received its initial charter prior to the finalization of the plans for 
NALDA Phase I, although funding for LMDSS was and still is received mainly under 
the NALDA system. According a phone conversation with the Operations/Planning 
Department Head for NAVAIR 3.6.2, the LMDSS system has always been funded out of 
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excess year end funds and primarily from the NALDA program. For the last few years, 
LMDSS has been funded "out of hide" from NAVAIR 3.6.2 Operations and Maintenance 
(O&M) funds, according to the 1999 LMDSS Quality Assurance Leader. This constant 
scramble to obtain funding caused numerous hardships during the entire developmental 
‘process. 

In general, the NALDA AIS codes fall under Tab G of the Program Objective 
Memorandum (POM) 98/99 under AIS code 1274 (Naval Air Systems Command, 
Acquisition Program Baseline, 1997). The costs identified with the parent NALDA 
program included development, implementation and operating support costs. All funds 
for the development of software end with the start of FY00. Implementation funding for 
software is minimal until FY00, when it triples, but it should be noted that the total 
amount budgeted once the implementation starts is significantly less than in previous 


years as displayed in Table 1 below. The APB also stated that the year-end funds are 


often used to enhance the upgrade and modernization schedules. 





FY 1999 FY 1999 FY 2000 FY 2001 


SW 3,758,000 1,844,000 

Development 

SW 250,000 459,000 681,000 693,000 
Implementation 


Table 1 NALDA Costs excerpted from NALDA Phase H Acquisition Program 
Baseline, 1997. 


At this time, the NALDA program has missed milestones and 1s not receiving any 
additional funding. LMDSS is still not functioning as it should and all additional 
developmental funding is being redirected to getting the LMDSS application running 
smoothly. This is hampering the developmental efforts of other systems (Phone 
interview Jahn/Evanoff, 1999). 







32 








=F Current LMDSS Scheduling Plans 


Under the new APB, LMDSS developers now had to meet the deadlines 
established for the Milestone III Decision Authority. The first increment, which 
consisted of the consolidation of the configuration management and aviation 3M 
systems, was to be completed by February 1998. Figure 2 displays how the completion 
of LMDSS is an integral part of the NALDA system (NAVAIR 3.6, 1999). In the 
process of transitioning from one software language and platform to another and to meet 
the goals of the NALDA program, many DSS items were discarded. Graphics 
capabilities and some of the statistical forecasting and modeling modules were scrapped. 

One of the major setbacks is the lack of preformatted reports for the system. 
Nearly 50 reports were initially developed and coded but since the structure of LMDSS 
changed it has forced the continued running of the Mechanicsburg site in order to ensure 
continuous exchange of data with the Fleet. The question NAVAIR is now trying to 
answer is should it either pay to update the praia coding, or take a different attack and 
develop a structured query language based means of obtaining the report forms. 

Throughout this transition, the LMDSS program was plagued with other setbacks 
which resulted in more time delays: 

@ A Product Support Team member who pushed hard to transition to the 


HTML format was transferred taking with him a lot of corporate 
knowledge. 


@ The merging of databases resulted in some corrupted data. 


@ A hacker successfully attacked the LMDSS server corrupting invaluable 
data and resulting in LMDSS being taken off line to develop additional 
security measures such as firewall integration. 


@ Further testing revealed LMDSS's Active X programs could not be 
downloaded to users with certain web browsers who were not accessing 
the Internet with at least a T-1 connection. 


e The resulting decision to abandon ActiveX for Java caused additional 
delays due to training deficiencies and data reliability problems. 
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In 1997, proposed performance measures were written for NALDA Phase IL. 


Areas mentioned in this document related to the LMDSS system were: 
° User satisfaction 
e Logistic support costs 


® Data accuracy and timeliness 


Unfortunately, this document did not accurately outline specific performance 
measurements that could be used in the developmental or implementation phases of the 
LMDSS project. For example, the User Satisfaction section proposed that the 
effectiveness of the act of combining the NALDA II subsystems be measured through the 
use of a survey conducted six months after the technical architecture was implemented. 
Although a functional requirements document was developed, it was not available for 
review. It is possible that more specific measures were developed in this document. 


(Naval Air Systems Command, Performance Measures, Proposed, 1997) 
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IV. _ METHODOLOGY 


A. RESEARCH QUESTIONS 


The primary research question is: "Does LMDSS meet user requirements?" The 


subsidiary research questions mentioned earlier are: 





How can LMDSS be improved to better meet managerial needs? 

What are the significant decisions users would like LMDSS to support? 
How has LMDSS been funded? 

How can NAVAIR improve the LMDSS program? 


What can future software developers learn from studying the LMDSS 
project? 


What are the key attitudes and opinions of LMDSS users? 


The investigation into the last of these questions led to the additional difference 


(group comparison) questions below: 


Do military and civilian data analysts at the various organizational levels 
(fleet command, type command, system command, etc.) have significantly 
different LMDSS user requirements? 


Do various categories of users (officers, enlisted, General Schedule--GS, 
civilian) have different LMDSS requirements? 


Is there a difference between the attitudes of LMDSS users at various 
organizational levels concerning modeling/simulation, graphics, exporting 
data, and the ad-hoc query (IQ) tool? 


Is there a difference in the frequency of LMDSS use among the LMDSS 
organizational groupings? 
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B. DESIGN OF THE STUDY 


This research is best described as an ex-post facto design consisting of a one-shot 
questionnaire survey (Campbell & Stanley, 1963) triangulated with face-to-face and 
phone interviews. The term ex-post facto 1s an appropriate description because the 
research team did not control or manipulate the independent and dependent variables 
identified. The research conducted on the LMDSS project was exploratory or pre- 
experimental in nature. Simple random samples were chosen from a population of 843 
LMDSS users registered with NAVAIR 3.6.2. A Web-based survey, face-to-face 
interviews, and phone interviews were the methods used to collect data. Past NAVAIR 
LMDSS briefs and documents posted on the NAVAIR 3.6.2 Web site were secondary 
sources of data (NAVAIR, May 1999). 


C. CONDUCT OF THE STUDY 


I. Research Strategy 


a) Prestudy 


The prestudy phase of this research began in November, 1998 during a 
three-day visit to NAVAIR 3.6.2 at the NAS Patuxent River, Maryland. Fifteen personal 
interviews were conducted with an emphasis on questioning those individuals directly 
involved in the management of the LMDSS project. Most of the interviews were 
recorded and notes were used to document the minor discussions that took place. An 
exploratory focus group was held on the final day of the visit with the three individuals 
assigned to the LMDSS Help Desk. The purpose of the focus group was to gain 
additional insight missed during the interviews and help develop meaningful questions 
for the forthcoming survey. The focus group was moderated by the thesis authors and 
videotaped. See Appendix C for a complete list of the questions that were asked. An 
Access 97 database containing the total population of 843 LMDSS users was received 
from NAVAIR during this visit. 


38 











Upon leaving NAVAIR, the research team traveled to NAS Norfolk and 
NAS Oceana completing five personal interviews with LMDSS users at the Safety 
Center, Aviation Intermediate Maintenance Depot (AIMD), Wing, and Squadron levels. 
These interviews were exploratory in nature and were documented in the same manner as 


those conducted at NAS Patuxent River. 


b). ~The Pilot Study 


The pilot study phase consisted of choosing two random samples from the 
LMDSS user population, developing and pilot testing phone interview and survey 
questions, validating email addresses for users within the samples, mastering the web 
survey software, and setting up a web site to host the survey. 

Two systematic, ‘random samples (Cooper and Emory, 1995) were 
selected from the total population of 843 LMDSS users. A random sample of 289 
LMDSS users was selected to participate in the Web-based survey. A smaller sample of 
twenty-five LMDSS users was chosen to participate in the phone interviews. 

The phone interview and survey questions were developed from the 
information collected during the personal interviews and focus group conducted in the 
prestudy phase. The initial drafts of both the phone interview questions and the survey 
questions were emailed to the twenty-eight member LMDSS Quality Assurance Team for 
feedback. The QA team consisted of logistics and aviation maintenance experts within 
the NAVAIR domain who were selected by the LMDSS program manager to assist in 
testing LMDSS modules. The final drafts of both the phone interview and survey 
questions were developed from the email critiques and phone comments received from 
the QA group. 

The most time consuming task in the pilot study phase was validating the 
email addresses of the 289 and 25 member random samples. The LMDSS database 
received from NAVAIR had an abundance of entries dating back from two to four years. 
Of the original 289 LMDSS users selected to participate in the Web survey, only 213 
could be contacted by phone to attain their correct email addresses. All of the 213 users 
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contacted had access to the Web and were able to provide valid email addresses. These 
213 users became the effective random sample for the Web survey. 

The software application chosen to implement the web-based survey was 
Perseus Survey Solutions for the Web. Once the completed survey questions were 
entered into Perseus, they were transformed into a HTML format and posted on the Web. 
Coding within the survey was designed to send survey responses to a central Perseus 


server where they were encapsulated and sent back as email to the research team. 


Cc) The Study 


The study phase involved conducting the phone interviews, soliciting Web 
survey responses from sample members, documenting the phone interviews, collecting 
the survey responses, and sending out individual thank-you emails. 

Phone interviews, each lasting thirty to forty-five minutes, were conducted 
over a period of four months. Seventeen of the twenty-five member random sample were 
interviewed by phone as well as various LMDSS managers at NAVAIR 3.6.2. ‘As the 
research continued, additional questions arose that were not asked on the initial NAVAIR 
visit. Volunteer graduate students from the Naval Postgraduate School, who participated 
in related group projects, were tasked to contact members of the NAVAIR 3.6.2 staff for 
updated LMDSS information. The authors of this thesis developed the questions that 
were used and the answers received have been incorporated throughout this body of 
work. 

Initial emails requesting survey participation were simultaneously sent, or 
“shotgunned”’, to all 213 members of the effective random sample. As users replied, they 
were dropped from the shotgun list and sent thank-you emails. Email addresses were 
constantly changing throughout the study phase of the research. Even after all 213 emails 
. had been verified, at least ten to fifteen emails had to be corrected after each of four 
attempts to solicit additional survey responses from sample members. After the third 
solicitation attempt, approximately 120 of the 213 LMDSS users contacted replied. A 


final round of ninety-three individualized emails were sent to the remaining non- 
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_ participants requesting assistance in completing the survey. Thirty-two additional 
responses were received. After four weeks, a total of 152 LMDSS users from the 213- 
member sample replied to the Web survey. Individual thank-you emails were sent to 
each replying sample member within one day of receiving a response. 

Phone interviews were taped, with the permission of the interviewees, and 
later transcribed into notes. Only the interview questions developed during the prestudy 
and tested during the pilot study were used during the phone interviews with individuals 
from the twenty-five member sample. Additional sets of questions were customized for 
the each of the NAVAIR LMDSS managers contacted by graduate students throughout 
the study phase. Phone interviews proved difficult to complete because interview 
appointments were often cancelled at the last minute due to interviewee work conflicts. 
After months of interviewing it became more efficient to conduct the interview during 


the initial contact phone call instead of making future appointments. 


D. THE SAMPLE 
1. Random Selection Procedure 


The purpose of random sampling in this study was to attain samples of users that 
represented the characteristics of the larger population of LMDSS users. It can be argued 
that studying these samples enables accurate conclusions to be reached about the larger 
LMDSS user population. In order to strengthen this argument, a systematic random 
selection process was implemented to capture at least fifteen percent of the population to 
participate in the Web survey. Planning for a maximum non-participation rate of fifty 
percent, the initial size of the survey sample was approximately twenty five percent of 
the population. The systematic random selection process began when a seed number 
between one and eight was literally drawn out of a hat to select the first member of the 
sample. The seed number drawn for the larger of the two samples was a five. Every 
fourth user was then selected from the 843 LMDSS user database which was 
alphabetically ordered by last name. The resulting sample consisted of 289 users. A 
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similar procedure was chosen to select the smaller, second random sample of twenty-five 
names. The seed number randomly chosen was seven. The 289 user random sample was 
chosen to participate in the Web survey while the twenty-five user random sample was 
chosen to participate in the phone interviews. Additional Access 97 databases were 
developed to manage both random samples. (Cooper and Emory, 1995) 

It was assumed that the size of the twenty-five user sample chosen for the phone 
interviews was too small to enable precise estimations or inferences about population 
characteristics. The primary purpose of creating the smaller sample was to attempt to 
gain some additional insight by using open-ended questions that were more appropriately 
delivered by a phone interview than a survey. Time and workload limitations also 
influenced the decision to limit the number of random phone interviews. 


2: Validation of Email Addresses 


The NAVAIR LMDSS user database contained many old and outdated entries. 
This called for an aggressive effort to clean the data. Cleaning the user contact 
information involved validating the phone number, email, identity, and 
command/organization of over 350 users contained in the two random samples, the QA 
team, and managers at NAVAIR. During the period of this research, many commands 
transitioned to Information Technology for the Twenty First Century (IT-21) standards. 
This transition resulted in significant changes to email addresses throughout the Navy 
during the study phase of this research. Maintaining current email addresses on the users 
belonging to the two samples required an almost daily effort and proved much more time 
consuming than anticipated. Two to three phone calls were often required to contact 
each sample member. Many email addresses were retrieved only to change a few months 
later as Navy commands updated their networks to meet IT-21 standards. Table 2 shows 
the breakout of how the original random sample of 289 names chosen for the web-based 
survey decreased to 213 names as a result of the phone number/email validation. Three 
members who were contacted declined to participate because they believed their contact 


with the program was so minimal that they could not provide any relevant data. 
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Of the twenty-five LMDSS users selected to participate in the phone interviews, 
only seventeen were contacted. Researchers were unable to reach six of the sample 
members due to changes in both phone numbers and email addresses. The remaining 
two users failed to keep previously arranged appointments on several occasions and did 


not return subsequent phone messages. 


Reasons for Sample Shrinkage Number of LMDSS users 





Table 2 Shrinkage of Original Random Sample (Web-based Survey) 


3: Testing the Effective Sample for Randomness 


Due to the large attrition rate of the original 289-user sample, randomness could 
no longer be assumed. Additional testing was completed to ensure that the effective 
sample of 213 LMDSS users was still random in nature. Using the same systematic, 
random selection procedure described earlier (the random seed number selected was 
eight) a random test sample of 105 LMDSS users was created to be compared with the 
effective sample of 213 users. 

The demographic characteristics of both independent samples were measured 
based on data found within the NAVAIR database. For each sample, users were assigned 
to mutually exclusive and exhaustive user categories defined as: Officer, Enlisted, 
General Schedule (GS), Civilian, and Unknown. All members of both samples were then 


assigned to mutually exclusive and exhaustive organization types consisting of: Fleet 
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Command, System Command, Type Command, Program Office (PMA/APML), Depot, 
AIMD, WING, Squadron, Marine, Reserve, Private Company, and Unknown. The 
demographic data for both samples is depicted in Tables 3 and 4. 

The chi-square nonparametric test was chosen to compare the two independent 
sample groups because the demographic data as shown in the above tables was nominal 
and categorical. The requirements of independence between the groups, and mutually 


exclusive/exhaustive categorical data were met. 


@itruah Enlisted Civilian Unknown 


Effective 
Sample of 


213 users 


Sample of 
105 users - 





Table 3 Sample Demographics With Respect to User Category 


Organization Type Lffective Sample of 213 Test Sample of 105 







Users L-sers 


PMA/APML , 


a SE | 







ae a eR SAC RN 


Table 4 Sample Demographics With Respect to Organization Type 
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The null hypothesis notation is: H , : E; = T; , where Eis the effective random 
sample and T is the test sample. The null hypothesis claims there is no difference 
between the demographic characteristics of the effective 213-user sample and the 105- 
member test random sample. The level of significance (the probability of rejecting a null 
hypothesis that is true) is .05. Critical values of the chi-square distribution, based on 
degrees of freedom, were found in a statistical table from Cooper and Emory (1995, p. 
659). The chi-square tests comparing the samples, using SPSS statistical software, 
resulted in the output displayed in Table 5. 







Test Critical Value of Degrees of SPSS Calculated 
(.05 level of Chi-Square Freedom Critical Value of 
significance) Distribution Chi-Square 


User Category 
Comparison 9 49 4 3.461 
Between Samples 
Organizational 
Type Comparison 19.68 : 11 9.706 
Between Samples 


Table 5 Chi-Square Tests of the Two Independent LMDSS Samples 










In both chi-square tests comparing the user categories and organization types of 
the two samples, the SPSS calculated critical value was less than the critical value of the 
chi-square distribution. The null hypothesis therefore cannot be rejected. The samples 
are not significantly different at a significance level of .05. Since the effective sample 
and test sample were not significantly different, the argument can be made that the 


effective sample of 213 LMDSS users is indeed random and representative of the greater 


LMDSS user population. 


E. INSTRUMENTATION 
1. Need for a Custom LMDSS Survey 


Since LMDSS is a custom software application developed by NAVAIR for the 
Navy's use, there was not a commercial, off-the-shelf (COTS) survey instrument 
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available considered appropriate for the requirements of this research. The questionnaire 
developed by Moore and Snyder (1998) was designed for a different audience with a 
different research focus thus their instrument did not meet the needs of this research 
effort. COTS software was used to implement the Web survey (Perseus), manage the 
user samples (Access 97), and statistically analyze our survey results (SPSS v.9.0 and S- 
Plus v. 4.0). 

2. Developing and Pilot Testing the Survey Instrument 


In order to collect as many responses as possible from the 213 member effective 
random sample, it was decided to develop a Web-based survey. It was thought the 
novelty of a paperless Web survey would attract more responses and limit non-response 
error. Email was used to conduct multiple, rapid requests for user participation. 
Individually addressed emails seemed to generate a significantly greater response rate 
than group or shotgun emails to multiple users. 

The initial draft of the survey questions was created from the interviews and a 
focus group conducted at NAVAIR. The focus group was responsible for many of the 
questions relating to the quality of LMDSS training, the 1Q tool, and how LMDSS is 
used. Pretesting was accomplished by conducting a pilot test of the Web survey to detect 
any design weaknesses in the survey instrument. The pilot survey was posted on the Web 
and the twenty-eight member LMDSS QA team was contacted by email and requested to 
provide feedback. Thirteen survey responses were received. As. a result of these 
responses, the number of open-ended questions in the survey were reduced, several 
questions considered confusing were rewritten, and options such as: not applicable, 
unknown, and don't know were included. The final version of the Web survey is found in 
Appendix D. 

3 Pilot Testing the Phone Interview Questions 

The phone interview questions were developed in a similar manner as the survey. 
Initial draft questions were created using input from interviews and the focus group 


conducted at NAVAIR. The pilot questions were emailed to the LMDSS QA team for 
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evaluation. As a result of the QA team's feedback, several questions were added and 
several were rewritten for better clarity. The majority of phone interview questions were 
open-ended, to allow a level of probing and depth not considered possible by the Web 
survey alone. Fifteen of the seventeen phone interviews conducted were done by the 
same researcher to minimize interviewer bias. The questions developed for the random 


phone interviews are presented in Appendix E. 


F. INDEPENDENT AND DEPENDENT VARIABLES 


Many authors restrict the term independent variable (IV) to variables that are 
manipulated. Some researchers define an independent variable more broadly to include 
any predictors, presumed causes, or influences under investigation. Such independent 
variables that define attributes of persons or the environment and are not manipulated in 
the study are called attribute independent variables. (Morgan and Griego, 1998) 

The attribute IV chosen as a possible influence on the dependent variables 
measured in the Web survey was the organization type as previously defined in the -chi- 
square testing of the sample members. Cooper and Emory (1995) define an extraneous or 
moderating variable (MV) as a second IV that is considered because it is believed to have 
an influence on the original independent/dependent variable relationship. User category 
(officer, enlisted, General Schedule (GS), civilian) was suspected to be a MV. Morgan 
and Griego (1998) describe a dependent variable (DV) as the presumed outcome that 
measures the effect of the independent variable. An IV typically affects a DV in a way 
that can be measured. Surveys and interviews are two ways to measure such an effect. 

The DVs that were measured in the Web survey for possible IV effects to answer 


the difference based research questions were: 
e Importance of modeling and simulation 
e Importance of a graphics capability 
o Importance of a capability to cleanly export LMDSS data 


e Importance of a structured, ad-hoc query (IQ) tool 
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e Frequency of LMDSS use. 


G. ANALYSIS STRATEGY 


Research conducted with attribute variables is limited in its ability to establish 
proof of causality with any certainty. The standard of causation requires more complex, 
experimental research approaches where active, independent variables are manipulated 
over time. The scope of this research is limited to answering the main research questions, 
and exploring the aforementioned differences between the LMDSS user groups. 

Data from the phone interviews were transcribed from tape recordings made by 
the interviewers. The interview transcripts were analyzed to identify common user 
attitudes and perceptions. Trends were uncovered and compared with those found in the 
Web survey responses. 

S-Plus statistical software was used to conduct a frequency analysis on the closed- 
response (specified alternatives) questions within the Web survey. Content analysis was 
completed on the responses provided from the open-ended survey questions. One-way 
Analysis of Variance (ANOVA) testing matching one of two possible [Vs (organizational 
type and user category) with each DV measured in the survey was conducted using SPSS 
statistical software. The one-way ANOVA was used to assist in answering the difference 
(group comparison) research questions. 

Open-responses, free choice of words, from the Web survey were collected and 
compared using content analysis with the other types of data to provide answers to 
‘specific research questions. Data from phone interviews with managers at NAVAIR 


3.6.2 was reviewed, summarized, and compared with other data sources. 
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V. FINDINGS 


A. ORGANIZATION OF FINDINGS 
The findings in this chapter are presented in the following sections: 
@ Frequency and content analysis 
e Establishing difference null hypotheses 
6 ANOVA tests 
@ Phone interview findings 


® Interviews with past and present LMDSS managers 


1. Frequency and Content Analysis 


The material in the first section, frequency and content analysis, includes a 
detailed analysis of the data collected from the Web-based survey. To assist in analysis, 
the survey questions were grouped into the following subheadings according to the topic 


covered in each question: 
e Demographics 
e Frequency of use 
® Hardware/software concerns 


e Usage of application 


® Modeling/simulation 
e Training 

e Graphics 

° User perceptions. 
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The presentation of the Web survey responses mirrors the approximate order of 
the questions as found in the web based survey questionnaire (See Appendix D). 

A variety of statistical software applications were used to conduct the analysis. 
SPSS version 9.0, S-Plus version 4.0 and Excel 97 were used to generate the charts and 
graphs contained in this chapter. A combination of summary tables, percentage 
histograms and figures were used to accurately display the findings for each survey 
question. 


2. Establishing Difference Null Hypotheses 


In the second section, establishing difference null hypotheses, the null hypotheses 
explored in this thesis are established based on the difference questions previously 
identified in the methodology chapter. These null hypotheses were developed for the 
difference questions derived from the last research question only. These hypotheses were 
tested via a one-way ANOVA strategy. The remaining research questions were explored 
using frequency and content analysis of the various data sources as previously discussed. 


3. ANOVA Tests 


In the third section, ANOVA tests, the parametric statistic known as one-way 
ANOVA was used to test the difference null hypotheses to assist in answering the final 
group comparison research questions. 


4. Phone Interviews 


The fourth section, phone interviews, presents a summary of the data obtained 
through seventeen phone interview surveys. This data collection tool was used to collect 
data not easily captured through the web survey. 


5. Interviews with Past and Present Managers 


To get a historical, managerial perspective phone and face-to-face interviews 
were conducted during the course of this research project with past and present LMDSS 


managers. A summary of these findings is contained in the final section of this chapter. 
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B. FREQUENCY AND CONTENT ANALYSIS 
1. Demographics 


The two IVs considered in this thesis are the user categories and the various 
organization types. User category demographic data will be referred to as IV-1 and the 
organization type as [V-2. 

The user category groups (IV-1) were broken into four subsets: Officer, Enlisted, 
General Service (GS) and Civilian. IV-1 demographic data was obtained through the use 
of the original NAVAIR LMDSS user database. A result of the breakdown of the 
respondents is displayed below in Table 6. 


Group Name Number of Responses Percentage of 152 


Respondents 


General Service 


Table 6 User Category Groups (IV-1) Summary Results — 





The organization type was obtained through direct input by the survey 
respondents. The resulting organizational categories are listed in Table 7. The data 
found in Table 7 shows a majority of the responses came from survey members attached 
to system commands, civilian firms under contract to the Navy, and depots. A total of 65 
percent of the total survey response data was received from personnel in these three 
organization types. The two organizations, whose members did not reply to the survey, 
Commander Naval Reserve Force (COMNAVRESFOR) and the Naval Safety Center, 
were automatically deleted from this and all following histograms to facilitate easier 
' analysis. A reason for the non-response from users within these two commands is not 


known. 
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Organization Type Number of Percentage of [52 
Responses Respondents 


Systems Command 40 % 
(COMNAVAIRS YSCOM, 
SPAWAR... 


26 
24% 





a ae 
ee 1 
ee ae ee eee 
ae 
Type Command (COMNAVAIRPAC, 2) 3% 


COMNAVAIRLANT... 


[Marine CorpsActiviy =| 8 


CINCPAC, COMTHIRDELT... 
[Naval SafetyCenter | OH 
[COMNAVRESFOR | OH 
Total 100% 


Table 7 Organization Type Groups (IV-2) Summary Results 


2. Frequency of Use 


Survey questions 1, 3 and 4 as found in Appendix D are covered in this section. 
For ease of use, each figure in this chapter containing a percentage histogram also 
displays the specific question as asked in the original survey. 

Figure 3 displays the percent histogram for question 1, “How often did you access 
LMDSS when it was available on the web?” It shows less than 15 percent of all 
responding users accessed the LMDSS program on a daily or weekly basis. Over 41 
percent used the application infrequently and 26 percent answered Not Applicable. This 
may partially be attributed to the long period of time that the program was frozen. New 


users were still being issued passwords but were unable to access the system. 
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Percentages 


40 
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20 
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Daily Dont know iInfrequentiy Monthly WNot.applicable VVeekly 


Of How often did you access LMDSS when it was availabie on the Web? 


Percentages 


Figure 3 Frequency of Usage Percent Histogram 








1-2 years 3-4 years Dont remember More than S years 
2-3 years 6-12months LessthanSmonths Not appicabie 


Q3 How tong hes it been since you lest acceseed LMDSS? 


Figure 4 Duration Since Last Access Percent Histogram 








No 
Oo 


Percentages 


> 
So 





Q4 When were you first introduced to LMDSS? 


Figure 5 Length of Time Using LDSS Percent Histogram 


Questions 3 and 4 are directly related to each other and are covered in Figures 4 
and 5. Question 3 asked, “How long it has been since you last accessed LMDSS?” and 
question 4 queried, “When were you first introduced to LMDSS?” From Figure 5, it 1s 
apparent that over half of the users signed onto the system in the last two years yet only 6 
percent of respondents have signed on in the last six months. 


3. Hardware/Software Concerns 


Results of questions 5 through 8 are discussed in this section. Figure 6 indicates 
the findings of question 5, “When the LMDSS Web site comes back online, what will be 
your primary means of accessing it?” As shown by the percent histogram, nearly 60 


percent of all respondents reported they accessed LMDSS through the network. 
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6 
Dont know Other Vie the network 
56K modem ISDN connection T-1 : 
Q6 When the LMDSS eite comes beck on line, what will be your pranay means of eccesing it? 


Figure 6 Primary Means of Access to LMDSS Percent Histogram 


Question 6 asked the respondent to check all answers that applied to the 
following question: “What kind of problems did you have accessing LMDSS?” Results 
are tabulated in Table 8. 


Answer Number of Percentage of 152 
Responses Respondents 


| have not had any problems 


Don’t know, just could not access the 21% 
ae web site 


pOther | 8H 
[Firewall blockingaccess | 28S 
‘Intemet browserissues | 
[Proxydifficulties 
| Total without difficulties: | S| 84 
| Total with one or more difficulties | 101 | 66% 
Total with two difieutties {7 fe 


Table 8 Type of Access Difficulties Table Summary 
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It should be noted that although 101 people reported at least one difficulty, only 
twenty-eight reported encountering more than one difficulty. On the other hand, forty- 
two percent of the respondents responded with “don’t know” or had some “other” type of 
problem. Without knowing the experience level of the users and the specifics of their 
inability to access the site, it is impossible to determine exactly what type of difficulty 
these respondents encountered. They may have encountered unique problems or they 
may simply have been unable to identify the problem. 

Table 9 presents the cross tabulation summary statistics between IV-2 
(organization type) and the type of difficulties encountered by the surveyed users. It is 
interesting to note that civilian firms appeared to encounter firewall difficulties at a 
greater rate than their government counterparts. Twenty-five percent of all civilian 
contractors surveyed and nearly twenty-two percent of all depot respondents reported 


firewall trouble. Systems commands, which were the largest of the surveyed 


organizations, also reported firewall difficulties but at a much lower rate of 7 percent. 







Type of Difficulty Top Three Organizations and 
and Total Number Number of Difficulties per Or ganization 
Firewall 


a aad a 
| TYCOM | 

ial 

matrix tables 

the IQ tool 

eae ee 

inadequate 

Internet browser | 8 7 NA | 

ic ee ee | 3[Depot | 21 TYCOM | 1 


“ee Eee ee iy 
Other 
Depot 
=D 
| SYSCOM Other 


Table 9 Summary of Access Difficulties by Type Organization 
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In addition to the firewall access problem, a lot of users answered with “other” or 
“don’t know”. Not surprisingly, the three organizations that comprise the largest 
percentage of the total survey respondents (systems commands, civilian firms and depots) 
reported the largest number of “other” and “don’t know” responses. 

Although the systems commands provided the greatest number of respondents, 
many of the more specific access problems were identified by the depot level users who 
lodged a greater quantity of complaints. Part of this might be attributed to the fact that 
depot users provided the largest number of complaints concerning inadequate computer 
hardware. There may be a correlation between inadequate hardware and the access 
problem but further study would be required. 

Question 7 asked, “When LMDSS is back online, users will be required to use 
Netscape Communicator 4.5 with 128 bit encryption. Do you anticipate any problems 


downloading and installing this browser?” Figure 7 presents the results of this question. 


Percemages 








CG 


Dont know No Not applicable Yes 
Q7 When LMDSS ie beck online, ueers will be required to uee Netecape Communicator 45 
with 12@ bit encryption (aveilable for free downioed et wvew.netscape.com). Do you anticipate 
any problems downloading and installing this browser? 


Figure 7 Browser Installation Percent Histogram 


As shown in this question result, there does not seem to be any anticipated 


difficulty in the change of browsers. 


57 





Question 8 was an open-ended question that queried users on “What type of 
software problems did you experience while using LMDSS?” Table 10 summarizes the 


results of this query. 


Category Number of Responses Percentage of 152 








Respondents 








| 


* Seven users who reported no problems also entered comments. 





Table 10 Summary of Respondents Reporting Software Problems 


Although only 17.8 percent of all respondents entered comments about the 
problems they experienced while using the LMDSS application, many of those who did 
reply entered multiple areas of concern. Several themes of dissatisfaction were easily 
identified because they were repeated throughout the commentary. The top five concerns 


raised in the dialog from the question 8 submissions were as follows: 


@ Inability to access LMDSS 


® LMDSS was not easy to use 
e Report generation was not available 
® Data was not available or unreliable 


® Netscape vs. Explorer 

Below are some of the more insightful comments on the aforementioned top five 
concerns. These comments do not necessarily reflect the opinions of the authors and are 
recorded as depictions of the beliefs of the survey respondents. 

On the inability to access LMDSS, several reasons for the difficulties were listed 


but the next two comments best characterize the major difficulties encountered. 
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Just couldn't access [LMDSS]. When I did it stated ‘Server Down’ or 
another time it said that LMDSS was in the middle of moving...Referred 
me to yet another site and it stated it was not available... 

Civilian Firm - Senior Systems Analyst 


FIREWALL! Could not fully access LMDSS when working off-base. 
Civilian Firm - Analyst 


On the ease of use, the comments varied from specific to broad in scope. The 


three examples below provide a sense of the overall picture. 


Too difficult to gain and maintain access for occasional users. If you don't 
use it frequently, you end up not being able to use it at all. 
USMC - LTCOL ASL-33 


Navigation to the screens I was interested in was NOT intuitively obvious. 
By obvious, I mean that a relatively inexperienced user could not locate 
the correct link (page) with the first quick scan edit. I was looking for 
SALTS information. 

Systems Command-NAVAIR Logistics Info Systems Engineer 


... The method of data extraction and the flexibility of output data format 
are no match for the current NALDA/ECA output. 
Civilian Firm - Engineer 


On report generation, several systems command personnel complained that 
advertised report functions were not available. One gentleman from a helicopter wing 
not only had difficulty using the canned reports, but had greater difficulty trying to use 
the IQ tool. He wrote: 


Although I attended the IQ class here at NAS North Island from a billet 
assigned by AIRPAC, I was never able to get the IQ program to work. 
Contacted CNAP personnel and they were having the same problems. 
The support at PAX River provided by (name withheld) was useless at the 
time also. So I just gave up until it is fixed. I do need it and reverted back 
to doing reports from the old NALDA and TDSA databases.... 

Wing - Configuration Manager 
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Other personnel reported they did not use LMDSS in the past due to the 
unavailability of data for certain aircraft type or components or that data they did receive 
was unusable due to unreliability or incompatibility with older systems. 

Lastly, on Netscape versus Explorer, although question 7 results indicated that 
there is little anticipated difficulty foreseen in getting and installing the required browser, 
there are still some questions about the decision itself. Some respondents questioned the | 
reasoning behind the move to use Netscape. One squadron maintenance Chief with 


extensive computer experience wrote this statement: 


I'd note here that the above requirement for Netscape is poor headwork. 
Put aside that it is not in keeping with the IT-21 "Standard" that has been 
endorsed by CINCLANT and CINCPAC...it becomes an administrative 
and logistical burden to marry oneself to a particular browser.” Squadron 
Data Analyst 


4. Usage of Application 


This section focuses on how the respondents have used LMDSS in the past and 
how they intend to use the application in the future. The survey questions that will be 
analyzed in this section include questions 9-14, 16, 18-19, and 35-37. Questions that 
affect the IQ tool will also be covered here. | 

The first two questions in this section were open-ended. This was an intentional 
tactic on the part of the authors to illicit unbiased responses reflecting what the users 
_ actually want from the LMDSS program. Although a list of choices would have been 
easier to tabulate, it is limiting in nature. Question 9 was, “What kind of information do 
you expect to retrieve from LMDSS when it becomes available?” Table 11 summarizes 
the respondents’ entries on this question. 

Comments received from this question were more difficult to analyze and 
categorize than previous questions due to the extreme range and variety of submitted 


answers. As before, many responded in vague generalities while other responses 
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precisely explained what types of data they desired to receive from LMDSS. Seventy- 


nine users entered comments for this question. Many of the comments contained more 


Category Number of Responses Percentage of 152 






Respondents 


52 % 
Don’t Know a es et 
[Not Applicable | HTH 
DonotintendtouseLMDSS {oo 2 
‘Leftcompletelyblank | 2 

















Table 11 Summary of Respondents’ Expectations About LMDSS Information 






Comment Categories Number of Percentage of 
Responses 152 Responses 


Reliability and maintainability data 
-All3M dataforallnavalaircraft (ss | FTC 
‘Trendanalysisdata CT 


Historical maintenance man-hour ot 
Repairable histories 

Flight hour summaries 

Technical directive status information 

All data currently available in NALDA I 

Cross reference tables for WUC, P/N, NIIN, UI 
Historical SRC data 

Support equipment historical data 


Table 12 Summary of Respondents’ Comments About Information Available from 
LMDSS 


















o) 






than one desired type of information. Table 12 provides a frequency breakdown of the 


user requested information type categories. 
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When available, the command organization and job title were included from the 
respondent database to provide insight into why these specific requests were made. 
These comments are not all inclusive but were representative of the sample. Some 


specific comments received from the responses to this question are listed below: 


Cannibalization data, supply data, NMCS/PMCS data. 
Wing - Data Analyst 


Would like to track failure and removal data for existing systems and new 


system applications during initial installation and subsequently as a 
fielded system. 


Systems Command - PMA Data Analyst 


Failure data of a particular part number. Hopefully this information will 
help me decide if we want to open up an engineering investigation when a 
failure is reported by the Fleet. 

Depot - Data Analyst 


Primary concern is failure data as it relates to Depot repair. 
Systems Command - NAVAIR Data Analyst 


Question 10 asked, “What information do you require but are unable to access via 
LMDSS?” Table 13 summarizes these results. 


Category Number of Respondents Percentage of 







Respondents 





Table 13 Summary of Respondent's Requirements from LMDSS 








Fewer people answered this question than the previous one. Referring back to 
Figure 3, the Frequency of Usage percent histogram, only thirty-seven out of 152 
respondents reported using LMDSS on at least a monthly basis. It is not unexpected 


therefore to find that a large percentage of users do not know what LMDSS can and 
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cannot provide. The length of time LMDSS has been inoperable is also a contributing 
factor toward the low response rate for this question. Numerous statements reflected the 
fact that LMDSS was not fully functional nor were its exact capabilities well known by 
the users. Some items requested by the users responding to the web survey are already 


incorporated into LMDSS such as: 
® Historical data by part number for all Navy Aircraft 
e Current Aircraft Inventory Records for specific type aircraft 


e OP-20 Data, AH-1W Maintenance Data, UH-1N Maintenance Data 


Others requested data that is currently contained in other portions of the NALDA 
system. For example: “Info available on NALDA I ECA report 733” was one comment. 
Still others were unclear on whether LMDSS would have the correct capability but made 
suggestions on what information they believed was needed. Due to the variety of the 
requests, categorization by subject was not possible. Below are some of the more 


articulate comments in random order: 


We need failure information at the serial number level with the JCN 

linked to depot data showing the actual repair done. Also need link to 

aircraft configuration information showing the number of cycles (hours, 

landings, etc) that the component lasted between last repair and failure. 
Depot - R&M engineer 


Work Unit Code (WUC); NIIN Cross reference top degraders by cost 
reliability and supportability mean time between failures (MTBF) for 
WSC (Actual and Predicted) cost of maintaining WUC service record card 
information. USMC - Depot Logistics Management Specialist 


I sure could use real time AIMD engine inductions and current status. 
Also real time bare firewalls. Although AEMS gives me this info, I would 
like one central database. Maybe LMDSS already does this, but only a 
few engine areas were up and running. 

Type Command - COMNAVAIRRESFOR PP Logistics Manager 
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History on components, aircraft, engines. Data to use as analytical 

processes to determine extent of work within the depot. Review RCM 

data throughout the responsible commands in the DMD reporting chain. 
Depot 


Haven’t been able to sort or select by unit identification code (UIC) or 
command, Secondary data: Type-Model-Series (TMS) of aircraft, work 
center, WUC, labor hours, beyond capability of maintenance (BCM) 
hours, etc... 

Fleet Command - CINCPACFLT Manpower analyst 


I would like to have information on the current state of the Naval Aviation 
Fleet in terms of: corrosion, fatigue, cracking, engine failures, accidents, 
corrosion prevention programs... 

Systems Command - NAWC Electrical Engineer 


Cost per flight hour, mean time to replace with valid metrics (PMB issue), 
mean time between failures with valid metrics (PMB issue) 
Type Command - COMNAVAIRPAC 


Question 11 asked, “How difficult was it to build matrix tables within LMDSS?” 
The question was done in a five-point interval Likert type scale with a score of one 
identified as “impossible” and a score of five being classified as “easy”. Results are 
shown in Figure 8. All Likert formatted questions are displayed in three dimensional 
graphs. | 

As shown in Figure 8, the majority of the respondents answered Not Applicable. 
This could be attributed to the fact that the LMDSS program had not been on line for a 
while or that the users had not used the tool to build matrix tables.. This could also be an 
indicator that more training needs to be conducted on this aspect of LMDSS’ capabilities. 


The results of those who did respond are tabulated in Table 14. 
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Figure 8 Matrix Build Difficulty Level Percent Histogram 


Category Number of Responses Percentage of 41 
Respondents who 
Answered 
12% 


Total who answered 


Table 14 Matrix Build Difficulty Likert Results 


Most people responded to this question by answering in the middle of the five 
point Likert scale but more tended to lean towards difficult than easy. Fewer than five 
percent of those who answered selected “easy”. 

Question 12 asked the respondent to check all answers that applied to the 
following question, “Which LMDSS modules did you access the most?” Results are 
tabulated in Table 15. 
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Answer Number of Percentage of 
Respondents Respondents 





Other a ee ee © 
Feature Synopsis ee 


Total who used at least one area 
Total who Don’t know or NA a ee ee <1 ee 


Table 15 Type of Access Difficulties Table Summary 


Percentages 
r=) 8 r=) 8 


(Not Answered) Dont know No Not applicable Yes 
013 Have you ever used the [Q tool to build any custom queries? 


Figure 9 IQ Tool Usage Percent Histogram 
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The next three questions to be analyzed, questions 13, 14, and 16, all focus on the 

IQ tool. Question 13 was a straightforward question that asked, “Have you ever used the 

IQ tool to build any custom queries?” Results are shown in Figure 9. As displayed in the 

percent histogram, the majority of users had not used the IQ tool. Only eleven people 
surveyed (7 percent) reported using the IQ tool. 

| Question 14 asked “Do you think the IQ tool was user fnendly?” This question 


had a total of sixteen respondents (11 percent) answer with a yes or no response and the 





8 


Percentages 


8 





(Not Answered) Bont know No Not applicable Yes 
Q14 Do you think the Q tool was user friendly? 


Figure 10 User Perceptions of IQ Tool’s Ease of Use Percent Histogram 


results are shown in Figure 10. Only two percent of the IQ users felt the tool was 
fniendly. | 
Question 16 differs from the two previous questions in that it asks about the users’ 
beliefs on the importance of including an IQ tool within the LMDSS application. 
Question 16 asked, “How important do you consider the addition of a structured ad hoc 
query tool to LMDSS?” Results are included in Table 16. Figure 11 displays the percent 


histogram for question 16. 
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Category Coding Number of Percentage of 152 
Responses Respondents 





Figure 11 Importance of IQ Tool Percent Histogram 


The majority (53 percent) of respondents declared the addition of an IQ tool 
would be “extremely” or “very important”. This is an interesting finding in light of the 
fact that only two percent of the respondents said that they found the existing ad hoc 
query tool easy to use in question 14 (Figure 10) and less than ten percent of the 
respondents stated they had accessed the existing tool in question 13 (Figure 9). 

Question 18 asked, “Please list a few of the key decisions you would like LMDSS 


to support when it comes back online.” This open-ended question was designed to focus 
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on the type of major decisions LMDSS needs to support. Most users described the type 
of information they required instead of describing the key decisions as requested. Table 


17 summarizes the results of this query. 







Category Number of Responses Percentage of 152 


Respondents 
Not Applicable 


Left completely blank | 


Table 17 Summary of Respondents’ Recommendations on LMDSS Key Decisions 














Due to the large variety of answers, categorization of this data was not practical. 


Four insightful comments listed below are representative of the collected results. 


Life cycle cost decisions, reliability centered maintenance decisions, 
integrated maintenance concept decisions, affordable readiness decisions, 
defense systems affordability decisions, supportability issues, acquisition 
issues, etc... 

Depot - Mechanical Engineer/Data Analyst 


Create some simple tool to show, in plain language, the present support 
status of a given item. One page with cost, AVDLR, items on hand, items 
due in, quarterly demand, number of maintenance actions at 
Organizational and/or Intermediate level for the most recent twelve 
months, and number of A-799s within those maintenance actions should 
give the typical user (and his/her novice level management audience) the 
sufficient data to make educated decisions. 
Depot - Electronics Engineer 


Part number usage and part number cost of ownership (how much does it 
cost to maintain a part number over a specified time). 
Civilian Firm 


Is end item experiencing infant mortality or wear out failure modes? How 
many cycles did the unit operate before failure? What was the specific 
failure mode? Is it worthwhile to invest in reliability improvements to a 
given end item (need cost and configuration data) 

Depot - R & M Engineer 
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Question 19 asked, “LMDSS will collect much of its raw data from NALCOMIS. 
If there is any data you will require which is not captured by NALCOMIS, please 
describe it.” Table 18 displays the summarized results. 


Category Number of Respondents Percentage of 








Respondents 


43 % 
[Enteredcomment | BB 


Table 18 Summary of Respondents’ Items not covered in NALCOMIS 







Many comments in this section reflected a lack of knowledge about the data 
available for retrieval within LMDSS. A few comments referred to 1tems not currently 


available within the LMDSS application. The other comments included such topics as: 
e Increased supply statistics 


e Support equipment data Technical Directives Library Configuration 
Control information 


e Government On-line Data System (GOLD) 


In general, most respondents appeared comfortable with the data provided by the 
Naval Aviation Logistics Command Management Information System (NALCOMIS), 
provided the databases and the matrices were properly formatted and accurate. 

Question 35 asked, “What might some of your reasons be for using LMDSS 
when it returns to the web? (Check all that apply)”. Results are displayed in Table 19. 

Question 36 was an open-ended question which requested users to clarify their 
| definitions if they checked the “other” category in question 35. Of the fifteen people 
who selected “other” in question 35, only ten entered comments. Three of the responses 


mentioned manpower calculations. Other selections are listed below: 
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Answer Number of — Percentage of 152 







Responses Respondents 


76 % 






decisions 
Search for data to complete periodic reports and 
forms 


Track the result of implemented improvements 


) i ae a 
Fiat aera ll al 
of system degraders 
|= whine Ml 
systems 
a 
engines 
ee 
platform readiness 
support costs . 
similar commands 


Other (Please describe in question 36 onl 
46 
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Completely Blank Entries 
Users who completed at least one block 


Table 19 Users Reasons for Using LMDSS 





Once again the reasons for using LMDSS justify the need for data 
analysis to review raw data and summarize info into a usable format for 
the APML.... Otherwise we will expend an awful lot of time on data that 
may conflict with other decision-making systems. 

Systems command - DAPML 


Get detailed failure mode information at the serial number level. Get 

depot data linked by Job Control Number (JCN) to maintenance action 

form (MAF). Get operating cycles information at the serial number level. 
Depot - R& M Engineer 
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Review historical hours and workload / backlog to assist in determining 
billet requirements for a commands Activity Manning Document. 
Fleet command - CINCPACFLT Management Analyst 


... I have had logistics people collect failure data of a particular part 
number to help in my decision to open up an engineering investigation on 
the failed item. 

Systems command 


Question 37 asked, “Referring to your answers in questions # 35 and # 36, what 
do you anticipate your primary reason for using LMDSS will be?” Results for this 
question are included in Tables 20 and 21. On this question some users repeated choices 
listed in questions 35 or 36 while others provided unique answers. Some of the replies 


from question 37 were: 


Category Number of Percentage of 152 


Responses Respondents 


Left completely blank 


Table 20 Summary of Respondents’ Expectations About LMDSS Information 
























Depends on training. Time, workload, and specialization are drivers. If 
training is intense: and concentrated, i.e. short term and Frequently Asked 
Question (FAQ)Help functions are improved, the primary reason for use 
will be to assist in the acquisition logistics determinations AND monitor 
those decisions in sufficient detail to allow for change before change 
becomes too costly. 

Systems Command - APML 


Since I have no data analysis support I would think this system if accurate 
could be of great benefit in managing my engine program on a day to day 
basis. 

Systems Command - Engine DAPML 
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Long-term analysis and development of improved analysis techniques (in 
particular, studying predictive indicators, similar in purpose to the 
Emergent Problems Matrix). 

Systems Command - Logistics Management Specialist 


...developing composite systems for comparative analysis, verifying 
system performance and cost once new acquisition is fielded. 
Other - ILS manager 


Answer Number of — Percentage of 1352 
— ee 






Support the Fleet 

Dmnill down to the MAF level to determine cause 
of system degraders 

Search for data to complete periodic reports and 
forms 

Measure the effect of improvement actions on 
support costs 

View cost data and flight hour history of aircraft 
engines 


Parts Availability/Tracking 


Compare the performance of my command with 
similar commands 


Table 21 Primary Reasons for Using LMDSS 


it 
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Measure the effect of improvement actions on support costs. This is vital 
to the Depots in reducing cost and continuing to produce a quality 
product. 


Depot - Maintenance Data System Coordinator 
5. Modeling and Simulation 


This topic was covered in survey questions 20 and 21. The former question was a 


Likert scale formatted question that requested the user to determine “How important do 


Category Coding Number of Percentage of 


Ve 


Responses 152 Respondents 


Table 22 Importance of Modeling/Simulation Capabilities 





you consider the development of a modeling/simulation capability -in future versions of 
LMDSS?” Table 22 and Figure 12 demonstrate the results of this question. 

From Table 22, it is obvious that nearly half of the users answered “don’t know” 
or “not applicable”. The majority of those who did voice an opinion were in favor of 
further development of modeling and simulation capabilities. This question yielded a 
much higher response rate than the similarly formatted question 11, Figure 9, which was 
based upon the IQ tool. This suggests a greater familiarity of modeling/simulation 


techniques among users than knowledge about the use of IQ tools. 
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Category Number of Responses 


Not Applicable 


Entered comment 


Did not repl 
Table 23 Response Summary for ModelVSimulation Examples 





Question 21 asked, “Can you provide an example of how a modeling/simulation 
capability to conduct sensitivity (what if) analysis will be useful?” Table 23 provides a 
frequency breakdown of the users who provided examples of modeling/simulations that 
would be useful. Many of the comments contained more than one example. The specific 


' types of examples are categorized and displayed in Table 24. 
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Comment Categories Number of — Percentage of 






Responses 152 Responses 


5% 





Predict future manpower and repair requirements _ 
and how they are affected by flight hour levels 


Impact of A799 removal levels on readiness 
pacts of aging 


Compare Intermediate level costs to Depot level 1% 
costs 












Determine return on investment for proposed 1% 
readiness and maintainability program changes | 
Justify Reliability Centered Maintenance (RCM) 2 1% 





and Engine Change Proposal (ECP) analysis 
Compare historic costs to actual projected costs 


Table 24 Summary of Types of Modeling/Simulation Examples 






Some of the more pertinent selections received from the responses to this 


question are listed below: 


For a low cost system like Joint Direct Attack Munitions, compare 

contractor depot vs. government depot vs. disposable, considering costs 

for movement, parts availability, labor, etc.... For a low cost container 

(approximately $600) determine the break point for defining ‘reusable’, 

considering costs of movement, repair, labor, materials, disposal, etc. 
Civilian firm - Logistics Analyst 


... perform some type of trend analysis or probabilistic modeling of the 
factors influencing engine repairs. This would help me build my engine 
repair requirements for the out-years of the budget. 

Fleet Command - Depot Requirements Officer 


...1f annual flight hours increase (or decrease), how many more 
maintenance man-hours will be required? If cannibalization actions are 
occurring, how many maintenance man-hours have been expended? 

Type Command - Analyst 


Life Cycle Cost Estimating - Sensitivity by changing anticipated flight 
hour usage. Impacts of aging - Sensitivity to cost increases of 
maintenance over time. 

Systems Command - Operations Research Analyst 
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We have to determine the trade-off of investment cost versus reliability 
improvement benefits. A model that would link future projected flight 
hours to a population of end items would be a very beneficial tool for this 
type of analysis. 

Depot - Readiness and Maintainability Engineer 


6. Training 


A total of five questions, questions 26 through 30, will be covered in this section. 
The first one, question 26, requested information on “How would you rate the quality of 
the LMDSS instruction you have received from NAVAIR personnel?” Results are 
compiled in Table 25. | 


Categorics Number of Responses Percentage of 152 






Respondents 


Bxcellent | 


Table 25 Quality of LMDSS Training 










Of the 152 respondents, only thirty-three had definitely received training on the 
LMDSS system and nearly fifty percent stated they had not had any training on the 
program. 

The next question built on this question and asked, “Did the training include 
Structured Query Language training using the IQ tool?” Of the thirty-three who received 
the training, nine answered “yes”, fourteen answered “no”, and the remainder replied 
“don’t remember”. Neither question probed into when the user received training so it is 
not possible to determine if the respondents who answered “no” took their training before 


or after SQL training was added to the NAVAIR syllabus. 
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Question 28 asked the users “Do you feel you will need LMDSS refresher training 
when LMDSS comes back online?” The majority of the respondents said “yes” (68 
percent) with only eleven people answering “no” (seven percent). The remaining thirty- 
four people (22 percent) checked the “don’t know” block. 

The following question was developed directly from the focus group that was 
held at NAVAIR. The focus group participants wanted to know the various levels of 
importance users assigned to different LMDSS training strategies. Question 29 was a 
Likert question that asked, “How important is ‘hands on’ LMDSS training with each 
student assigned to a computer terminal?” This question did not specify the location of 


the training or whether the training was to be a classroom environment or one-on-one 


training. Results are shown in Table 26 and Figure 13. 














Category Coding Number of Percentage of 











Responses 152 Respondents 
| 55 
Don’t Know, NA 





Table 26 Importance of Hands On LMDSS Training 


The numbers clearly show that most users place a high degree of importance on 
hands on training. 

The final question in this section was an open-ended question asking, “How can 
LMDSS training be improved?” A summary of respondents’ results is covered in Table 


27 and a list of means of improvements 1s listed in Table 28. 
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Figure 13 Importance of Hands On LMDSS Training Percent Histogram 


Category Number of Responses Percentage of 152 







Responses 


Table 27 Response Summary for LMDSS Training Improvements 







All respondents who stated that they had received training entered suggestions on 


how they would improve LMDSS training. User suggestions are categorized in Table 28. 


Several people made multiple suggestions within a single comment entry. 
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Comment Categories Responses Percentage of 






152 — 


More hands-on practice sessions with realistic data 
Develop an online Web based LMDSS tutorial 


bry eee I ll 
a more timely manner 

Extend training period to better cover material 4 | 3% 
Distribute CD-ROM tutorials 3% 
Develop two different training courses, one for — 


supervisors and one for data analysts 


Develop a more adequate “Help” function -—-2_ 













2 9 
] 2 
1 2 


2% 
1% 
1% 
1% 


More ad-hoc query training 
Expand training to include contractors 


Table 28 List of Recommendations to Improve LMDSS Training 


0 





Some specific comments received from the responses to this question are listed 


below: 


.. I have never seen any newsletters as to what is going on with the 
program as we did when there was just plain NALDA...I feel they have 
our email addresses and should put out a newsletter as to what is in the 
works, what some of the problems are, and when we can expect them to 
be resolved. 

Wing - Aircraft Configuration Manager 


LMDSS must be online when training is provided... When I took the class, 
LMDSS was not ready for us and in fact provided me with a rather 
negative perspective of its usefulness. 

Depot - Data Analyst/Engineer 


...l was quickly given an introduction to LMDSS during a trip to Patuxent 
River once, but never received training after getting my user ID and 
password. Had trouble getting in and eventually gave up trying. 

System Command - Program Analyst 


Tailor it for the level of user. As a PMA, I don’t anticipate using many of 
the intricate workings but I want access to check on things occasionally 
and I want it to be simple enough that I don’t become frustrated in the 
attempt. 

System Command - PMA 
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Additional training in the development and submittal of highly 
individualized queries. 
Civilian Firm -Engineering Support Analyst 


ge Graphics 

Questions 24 and 32 were developed to determine user concerns about graphical 
capabilities within the LMDSS program. Both questions were Likert scale questions but 
each focused upon a different aspect. The first question asked about the development of 
graphical capabilities within the LMDSS system; whereas, the second question queried 
the user about the need to develop the means to cleanly export retrieved data‘to already 
existing software applications. It should be noted that both of these two questions 
generated high responses. Question 24 had an 80.9 percent response rate and question 32 
received an 83.6 percent rate. 

Specifically, question 24 asked, “How important do you consider the 
development of a graphics capability in future versions of LMDSS?” The results are 
shown in Table 29 and Figure 14. In question 24, the largest group of users listed 
internal graphical capabilities as “somewhat important” but more users selected positive 


responses than negative ones 


Category Coding Number of Percentage of 






Responses 152 Respondents 


/Extremely Important [| 1 416% 
|VeryImportant | 8 
|Somewhat Important | 3 | SO 88 
|NotVeryImportant | 4 |S 10% 
[NotatAllIimportant | S| BH 


Table 29 Importance of Graphical Capabilities within LMDSS 













NO [ — 






A ff [Go 
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Figure 14 Importance of Graphical Capabilities within LMDSS Percent Histogram 
The next question asked, “How important is it for future versions of LMDSS to be 


able to cleanly export data to applications with graphics capabilities such as Excel?” 


Category Coding Number of Percentage of 


152 Respondents 
Extremely Important 
Very Important 


Table 30 Importance of Graphical Export Capabilities 
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Figure 15 Importance of Graphical Export Capabilities Percent Histogram 


This Likert question received the most positive response in the entire survey as shown by 
the data displayed in Table 30 and Figure 15. Nearly seventy percent of all respondents 
stated it was extremely important or very important to have good export capabilities to 
graphical applications. 

8. Users’ Perceptions about the LMDSS application 


This section contains the questions that were perhaps the most difficult to 
_ measure and analyze because they did not attempt to collect facts and figures. The 
questions asked here were used to ascertain the beliefs and perceptions of the users 
toward the LMDSS system. Since this type of information is normally difficult to collect 
using a survey tool, key questions were asked in several different formats: yes/no 
questions, Likert type scale questions and open-ended questions. Many questions were 
similar to try to obtain subtle nuances. The questions covered in this section include 


questions 15, 17, 22-23, 25, 33-34, 38-39. 
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The first perception question asked was a simple yes/no question inserted at the 
end of the Usage section. The straightforward question 15 asked, “Do you think LMDSS 
was user friendly?” Nearly half of the replies were in the “don’t know” category (49 
percent). Those who gave a yes or no answer were evenly split with forty saying “yes” 
(26 percent) and thirty-six saying “no” (24 percent). Again the large group of people 
answering “don’t know” may have been due to the extended period of time when the 
application was not in working order. Overall, these results suggest there is still room 
for improvement in this area. 

Question 17 asked, “If you know of other NALDA applications which contain 
query formats you would like to see implemented in LMDSS please list them below?” 


Table 31 summarizes the results of this query. 







Category Number of Respondents Percentage of 


Respondents 
Don’t Know or 
Left completely blank 


Entered comment . 












Table 31 Summary of Respondents’ Recommendations about Other NALDA Formats 


The only general theme appearing among most of these responses was simplicity. 
Users did not appear to hold strong opinions about query format as long as it was easy to 
learn and apply. The Aircraft Inventory and Readiness Reporting System (AIRRS) and 
the Aircraft Engine Management System (AEMS) were the only two NALDA 
applications that received at least three or more recommendations. One worthwhile 
comment came from an ILS manager from SPAWAR who suggested, “No NALDA 
applications come to mind, but you could apply the NSLC (shipboard) equivalent called 
"3M OARS". That application is very user friendly with ad-hoc reporting, conversion to 
commercial database software and web access.” 

Questions 22 and 23 were Likert type questions developed to assess how the users 


perceived the underlying structure of the software application program. The first 
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question targeted how much importance the user attached to knowing the structure of the 
report generation formulas and the second queried the usefulness of having an imbedded 
data dictionary. 

The results of question 22, “How important is it for you to know the details 
concerning how LMDSS reports are derived before you feel you can trust the information 


they provide?” are displayed in Table 32 and Figure 16. 


Category Coding Number of Percentage of 
— 152 Respondents 





Figure 16 Importance of LMDSS Report Derivations Percent Histogram 
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Nearly eighty percent of all respondents entered an opinion for question 22 and 
over fifty percent felt that it was “extremely” or “very important” to understand how data 
are derived by LMDSS in order to trust the data. This question does not address the 
reason why users believe this but it would be interesting to investigate the relationship 


between this question and the users reported inability to access the program reliably. 


Category Coding Number of Percentage of 
_— [52 Respondents 





Figure 17 Importance of LMDSS Data Dictionary Percent Histogram 
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Question 23 queried, “How important is a detailed description or definition (data 
dictionary) of all LMDSS data?” The results are displayed in Table 33 and Figure 17. 

Question 25 was another straightforward question asking, “Have you found the 
usage of definitions and metrics by LMDSS to be in compliance with established 3M 
definitions?” Since usage was measured to be low in Figure 4, a large number of 
meaningful responses was not expected. The results supported this expectation and are 


displayed in Table 34. 


Category Number of Responses Percentage of 152 
Respondents 





Table 34 Compliance with 3M Definitions Summary Results 


Question 31 requested users to “Please describe any examples of your not being 
satisfied with the quality of the data you have obtained through LMDSS." The summary 
results are provided in Table 35. Many of the comments contained more than one 


example. Table 36 provides a frequency breakdown of types of user answers. 


Category Number of Responses Percentage of 152 
Responses 


Not Applicable 


DontKnow | 
|Enteredcomment | kT 





Table 35 Summary of Examples of Dissatisfaction with Quality of LMDSS Data 
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Comment Categories Number of Percentage of 








Responses 152 Responses 


| Difficult to filter and obtaindesireddata | 6 | 4% 

Poor quality of 3M data from organizational level 
| No data for desired aircraft type, model, series | = 4 | 3% 
| LMDSS failure data did not agree with older systems | 2 | 1% 
| Cost per flight hour calculations erroneous | 2] 


Table 36 Summary of Respondents’ Comments 
















Some specific comments received from the responses to this question are listed 


below: 


Couldn't get the nght cut of engine information. It was — too hard to 
figure out how to get what I wanted. 
USMC - Maintenance Officer 


When data is normalized by flight hours it often is presented as all zeros... 
Systems Command - Logistics Management Specialist 


... was never able to get into many engine analysis areas. Half the area 
"was always under repair. 
Type Command - Logistics Analyst 


Was told (LMDSS) data was inaccurate, so did not use to a great 
degree... I did compare F404 engine/module removals with AEMS and the 
numbers were off (engine transaction report data base did not agree with 
your NALCOMIS 3M data base 

_ Civilian Firm - Engine analyst 


Depot maintenance data at the present time is inaccurate because of 
limited data being received from level 1 and 2 in the Depot... 
Depot - Maintenance Coordinator 


On the few occasions I've used LMDSS, the databases and info I needed 
were not loaded or current. 
Civilian Firm - Systems Analyst 
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I have no formal training of this system and found it impossible to use. I 
have visited six AIMDs and only two AZs were competent in drawing 
down data... CNAP maintains their own data base on site. 

Fleet Command - Management Analyst Manager 


To be honest, I got frustrated using the LMDSS system. It did not provide 
the information that I would need at the time so I usually built my own 
NALDA 82K queries and pull down my data in my that way. 

Depot - Analyst 


Questions 33 and 34 asked, “Has anyone from NAVAIR or the LMDSS 
development team ever contacted you requesting your input/feedback concerning 
LMDSS?” and “Have you ever attended a NAVAIR sponsored LMDSS users’ group 
meeting?” respectively. Twenty eight users (18 percent) responded that they had been 
contacted for input concerning LMDSS but only five people (three percent) reported 
attending a LMDSS users’ group meeting. Of those who attended, two were from depot, 
one was from a civilian firm, one was from COMNAVAIRRESFOR and one was from a 
systems command. 

The open-ended question 38 asked, “In what ways could NAVAIR make analysts 
more aware of LMDSS?” Table 37 summarizes the responses to this question. Fifty 
users entered comments for this question. Many of the comments contained more than 
one recommendation. Table 38 provides a frequency breakdown of how many users 


suggested each strategy NAVAIR could employ to make analysts aware of LMDSS. 


Category Responses Percentage of 152 







Responses 
‘Enteredcomment | SO | 8G 


Table 37 Summary of Suggestions on How to Increase Awareness of LMDSS 
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Comment Categories Responses —_ Percentage of 


152 Responses 














Periodic LMDSS user conferences 


Table 38 Summary of Types of Suggestions on Increasing LMDSS Awareness 


Some specific comments received from the responses to this question are listed 


below: 


If you build it they will come! 
NAVICP systems analyst 


Email notices... maybe like the NAVAIR Team Newsletters or the BPR» 
announcements that periodically show up in our emails. 
Systems Command - Operations Research analyst 


Most people have been aware of LMDSS or the lack of LMDSS for years. | 
If it’s up and running awareness will take care of itself. 
Systems Command - NAVAIR Deputy Department Head 


Get it up and running... Ensure data correctness. Let people access it and 
the word will spread if it’s good. 
Civilian firm - System Analyst 


Have a LMDSS user conference yearly. The group meeting could be run 
similar to the old NALDA user group meetings. Discuss problems, — 
concerns, and recommendations for a better system... 

Depot - Analyst 
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C. ESTABLISHING DIFFERENCE NULL HYPOTHESES 


In order to answer the difference, or user group comparison, questions described 


in the previous chapter, the following null hypotheses were developed: 


e H,: There is no significant difference between the user category groups (officer, 
enlisted, GS, civilian) with respect to the importance they assign to a 
modeling/simulation capability in future versions of LMDSS. 


e H,: There is no significant difference between the organization type groups (Fleet 
command, System command, Type command, etc.) with respect to the 
importance they assign to a modeling/simulation capability in future versions of 
LMDSS. | 


e H,: There is no significant difference between the user category groups in how 
important they consider a graphics capability to be in future LMDSS updates. 


e 4H, There is no significant difference between users in the various organization types 
concerning how important they view a graphics capability to be in future LMDSS 
updates. 


H,: There is no significant difference between user category groups with respect to 
the importance assigned to being able to cleanly export LMDSS data to 
applications with graphics capabilities such as Excel. 


e H,: There is no significant difference between users within the various organization 
types concerning the importance they assign to being able to cleanly export 
LMDSS data to applications with graphics capabilities such as Excel. 


H,: There is no significant difference between user category groups conceming the 
importance assigned to the addition of a structured, ad-hoc query tool to the 
LMDSS application. 


H,: There is no significant difference between the organization types concerning the 
importance assigned to the addition of a structured, ad-hoc query tool to the 
LMDSS application. 


e H,: There is no significant difference between the user category groups in how often 
LMDSS is accessed via the Web. 


e H,: There is no significant difference between the users within the various 
organization types in how often LMDSS is accessed via the Web. 
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D. ANOVA TESTS 
1. Meeting ANOVA Criteria 


A one-way ANOVA was used to compare the means of sample/groups in order to 
test for significant differences. ANOVA assumes the DV is interval scale, normally 
distributed, and the variances of the groups are equal. If the assumptions are violated, an 
equivalent nonparametric test called the Kruskal-Wallis (K-W) test can be used to 
compare the mean ranks of the groups. The IVs in this study are represented as ordinal 
data while the DVs were measured on an interval scale. The distributions of each of the 
five DVs measured were compared to a normal distribution curve to ensure they 
approximated the distribution of a normal curve. The equality of variances was tested 
using the Levene test. If the Levene statistic was significant (sig < .05), the assumption 
of equal variances was violated and the K-W test was conducted to compare with the 
ANOVA results. (Morgan and Griego, 1998) 

SPSS software provides a metric called the significance level, or sig, which 
allows the researcher to easily interpret a calculated statistic in a computer generated 
output without looking up the critical value in a table. Sig can be considered to be the 
probability of a Type I error. A Type I error is the likelihood of rejecting the null 
hypothesis when it is actually true. If the reported sig is less than .05 the finding 1s 
statistically significant and the null hypothesis (no difference) is rejected. (Morgan and 
Griego, 1998) 

The ANOVA statistic only compares means between groups and is used to 
identify a statistically significant difference between means. If the overall F value 
calculated by an ANOVA test is significant (sig < .05) then there is a statistically 
significant difference between groups with respect to the DV. Any further information 
comparing the means to each other is done with post hoc tests. Post hoc tests are only 


done if the overall F is found to be significant. 
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Zi ANOVA Testing Results 


ANOVA testing of each of the ten null hypothesis did not reveal any statistically 
significant differences among various user categories and organization type groups 
concerning the measurements of the DVs. In the three instances where the Levene Test 
revealed the violation of the equal variances ANOVA assumption, the K-W 
nonparametric test was conducted to see if a different result could be attained. In all 
three cases the K-W test results agreed with the ANOVA results. None of the null 
hypotheses could be rejected. 

Specific pairs of organization type groups were selected for ANOVA ‘testing to 
determine if a difference could be detected with respect to measurements of the DVs and 
none was detected. All thirteen organization types were combined into three groups: (1) 
higher echelon military commands, (2) lower echelon military commands, and (3) private 
contractors. Subsequent ANOVA testing did not reveal any evidence supporting the 
rejection of the null hypotheses of no differences between the groups. Table 39 provides 
a summary of the results of the one-way ANOVA testing conducted on the ten null 
hypothesizes. Since none of the ANOVA testing resulted in a significant difference 


between group means, post hoc tests were not conducted. 


E. PHONE INTERVIEW FINDINGS 


seventeen phone interviews were conducted from a randomly selected sample of 


25 LMDSS users. The major themes common to the majority of the interviews were: 
° Many expressed a strong desire for accurate and timely data. 
® Users want to be able to conduct effective ad-hoc queries on their own. 
© More training is needed on the use of the ad-hoc IQ tool within LMDSS. 


e The capability to drill down to the Maintenance Action Form (MAF) level 
is considered critical in investigating maintenance and logistics issues. 
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GP rarinicae 
H,, I value for ANOVA Sig K-W Sig Value Can II, be 
(IV and DV) ANOVA Value Rejected? 


User Cat and Levene Test not 
Modeling F(3,78) = .068 | . 977 significant NO 
Org Type and Levene Test not 
Modeling F(11,70) = .343 973 significant NO 
User Cat and Levene Test not 
Graphics F(3,120) = 1.15 334 significant NO 
Org Type and Levene Test not 
Graphics F(12.111) = .93 412 significant N 
User Cat and Levene Test not 
Data Export | F(3,148) = .96 412 significant Ni 
Data Export | F(12,139)= .92 529 496 
338 
340 
: 433 | 
=]. 316 





O 
: O 
| 
User Cat and rar ara Levene Test not me 
Ad-Hoc Tool | F(3,97) = 1.14 significant NO 
Ad-Hoc Tool | F(12,88) = 1.14 significant NO 
Freq of Use | F(3,97) = .922 507 NO 
Freq of Use | F(12,88)= 1.17 181 NO 


Table 39 ANOWA and Kruskal-Wallis Test Results 


° Most users interviewed had little experience or knowledge of models, 
simulations, and sensitivity analysis and were not interested in them at 
their level. 

® Analysts want to understand how data are derived to include any 


assumptions, definitions, and algorithms. 


e Most users interviewed expressed a lack of confidence in the accuracy and 
timeliness (two to three months late) of AV-3M data. 


e Many users expressed their frustration in knowing very little about the 
various data analysis software tools developed by NAVAIR. 


® Several users recommended teaching students at both the Data Analyst 
and A schools about LMDSS and other NALDA II applications. 
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® The majority of users recounted an unsatisfactory experience with 
LMDSS as a result of difficulties with queries and limited amounts of 
data. 


e Those familiar with the IQ tool found it extremely complex, requiring 
intensive attention to detail. 


e Most of the communication between the LMDSS users interviewed and 
NAVAIR occurs during sessions between the NAVAIR 3.6.2 help desk 
personnel. 


e None of the users reported being contacted specifically for user feedback 
concerning LMDSS. 


@ All users contacted had some form of a Pentium computer on their desk 
with a way of accessing the Web. 


e The majority of logistics analysts interviewed wanted the capability to 
track the parts much like FEDEX and UPS tracks shipments in the 
commercial world. Currently, finding the status of an ordered part, its 
location, and who controls its distribution takes a day's worth of phone 
calls to NAVICP. Several users on WING and TYCOM staffs considered 
this an important capability. 


Some statements recorded from those interviewed were as follows: 


Using LMDSS needs to be as easy as following the bouncing ball! 


Email is the best way for NAVAIR to stay in touch with customers at all 
levels. Email is a religion here. 


I'd like one big database in the sky which lets me choose the data elements 
I need. 


LMDSS is not very user friendly. I got lost trying to do queries. It needs 
to be more user friendly like Optimize NALCOMIS. 


I accessed LMDSS almost daily in the Fall of '98 for data queries. I was 
not aware the data within LMDSS was corrupted and the database had 
been frozen! 


I've served 15 years in the Navy at the organizational level and I never 
heard of NALDA until I got to the TYCOM staff. 
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LMDSS has not progressed enough to allow me to write my own queries 
like I did under NALDA I. LMDSS doesn't have the depth of the canned 
reports I accessed under NALDA I. 


We developed our own program called ‘Forecast’ which uses NAVFLIRS 
and comptroller info to conduct cost analysis, cost comparison, and 
degrader identification. Forecast can supply data within 10 days instead 
of the three months it takes LMDSS. 


I have been an active proponent of the technology change offered by 
LMDSS since its idea and conception years ago, but there is not any 
usable information there for me to provide my customers so I am not a 
user. 


F. INTERVIEWS WITH PAST AND PRESENT LMDSS MANAGERS 


During the period of this research, most of the managers within NAVAIR 3.6.2 
were interviewed both in person and over the phone. Some individuals no longer 
assigned to NAVAIR 3.6.2 were also located and interviewed by email and phone calls. 


The major findings of these interviews are summarized below: 


® No evidence could be found suggesting the use of automated software 
tools for documentation, test cases, defect tracking, and configuration 
management. 

® Little documentation exists which accurately describes the eight year 


history of the LMDSS project in a manner capable of supporting 
meaningful "lessons learned" training. There is a lack of historical 
software measurement data. 


@ A formal risk management program documenting threats and strategies to 
minimize them has never been conducted. 


® LMDSS has undergone several major requirements changes as well as at 
least four program manager changes. 


® Milestones and deadlines have been established and reestablished several 
times during the history of the LMDSS program. 


® The use of 8A contractors with limited technical capabilities and a poor 
understanding of LMDSS requirements resulted in years of delays and 
rework. 
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The LMDSS project 1s often funded out of hide and has never had its own 
line of funding. LMDSS has suffered more than one 50 percent budget cut 
during its development. 


Estimates of the time required, man-hours, and size of LMDSS related 
work was determined by dividing available funding by the cost of 
available, qualified programmers. 


Changes in programming languages required extensive rework of the 
canned reports. 


Most LMDSS requirements are developed by maintenance and logistics 
experts on the NAVAIR staff. A robust strategy of involving LMDSS 
users from the Fleet to provide feedback and input has not been 
implemented. 


97 


98 








VI. DISCUSSION AND ANALYSIS 


A. WHERE IS LMDSS NOW? 


In December 1998, LMDSS was taken offline to correct data error problems and 
load a minimum of eighteen months of AV-3M data for all types of Navy aircraft. The 
goal was to get LMDSS back online within a few months and complete QA testing by the 
scheduled NALDA I termination deadline of March 1999. This deadline was primarily 
motivated by a DISA mandate that all systems be Y2K compliant or terminated by March 
1999, Continuing high data error rates and problems loading the AV-3M data resulted in 
further delays in completing the LMDSS QA testing. Finally, NAVAIR 3.6.2 directed 
LMDSS be brought back online beginning May 1999 while QA testing continued. 
Although LMDSS is now back online and considered operational, only four aircraft 
platforms (E-2, H-60, S-3, A-6) are loaded as of August 1999. 

Currently, NAVAIR is continuing to struggle with loading the data into the 
underlying database structure. Data loads must be completed before all interface 
connections can be properly reviewed for integrity. Parallel operations with both 
NALDA I and UU servers are scheduled to continue during the months of September and 
October, 1999. In addition, there are three major short-term goals still unaccomplished 
by LMDSS developers: 


e Load at least eighteen months of data for all type equipment codes. 


e Successfully implement the automated update process using AV-3M data 
from the Streamline Alternative Logistics Transmission System (SALTS). 


e Reduce the high data error rate within the databases that feed LMDSS. 


e 
Until these objectives are accomplished, the usefulness of LMDSS as an effective 


management information system will be severely limited. 
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B. RESEARCH QUESTIONS 


This section applies the findings of our research in answering the research 


questions introduced in earlier chapters. 

1. What can future software developers learn from studying the LMDSS 

project? 

A well-documented history of the eight-year LMDSS project is important because 
it will provide invaluable lessons for future NAVAIR development teams. Many of the 
classic mistakes described in the Literature Review and Appendices of this thesis were 
identified during interviews with selected LMDSS program managers. The actual 
LMDSS software programmers currently contracted by NAVAIR were not interviewed 
during this research. Additional interviews with these programmers, their predecessors, 
and a few of the previous program managers not contacted will be required in order to 
capture a more complete LMDSS development history. 

The most significant factors that contributed to delays in the LMDSS 


development effort included the following: 
e Absence of a formal risk management program 
@ Major requirements changes 
° Lack of funding support 
© Proper contracting support 
® New technology 
@ Lack of user input 


° Personnel turnover 
A more detailed summary of additional software development mistakes that 


occurred during the LMDSS project is documented in section F of the Findings chapter. 
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a) Formal Risk Management 


A formal risk management program specifically addressing the LMDSS 
application was never implemented by NAVAIR during the history of the LMDSS 
project. The literature review described formal risk management as a proven, 
fundamental software development practice. The early identification of risks, evaluation 
of risks, and subsequent implementation of strategies to minimize the impact upon 


software development project is critical in the avoidance of major obstacles. 


b) Major Requirements Changes 


Transitioning LMDSS from a UNIX to a Web environment resulted in 
significant LMDSS requirement changes. A formal risk management program might have 
fostered additional communication and planning which would have allowed developers 
to be better prepared for the impact of these changes. | 

The majority of the major requirement changes experienced by the 
LMDSS program were responses to the exponential rate of improvements in networking, 
computers, and Internet technology within the last few years. Additional changes to the 
LMDSS project included the abandonment of the Ada and ActiveX codes for the newer 
Oracle and Java languages. Future development teams should study how these changes 
were implemented to learn the importance of proactive planning and estimating using 


formal risk management and software metrics. 


Cc) Funding Support 

LMDSS has suffered several major budget cuts during its history. 
Interviews with two previous LMDSS program managers revealed estimates of the time 
required, man-hours, and size of LMDSS related work were largely based on available 
funding instead of requirements. No evidence could be found suggesting that automated 
software tools, Computer Aided Software Engineering (CASE) tools, or standard 
software metrics were used to develop accurate estimates in planning and scheduling 


LMDSS work. As deadlines passed without LMDSS developers meeting stated goals, 
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credibility was lost with upper NAVAIR management. This loss of credibility negatively 
impacted the funding support received by the LMDSS program. 


d) Proper Contracting Support 


Interviews revealed Small Business Administration (SBA) 8A contractors 
were used to satisfy key requirements during the transition of LMDSS to the Web 
environment. On several occasions, the primary 8A contractor failed to deliver the 
desired software products citing misunderstandings ivolvine LMDSS requirements. 
More detailed requirements were drafted by the LMDSS program manager and accepted 
by the contractor. Subsequent attempts by the contractor to meet these detailed 
requirements were not successful. The primary LMDSS 8A contractor eventually was 


forced to go out of business and was replaced by a more qualified contractor. 


e) New Technology 


The new, complex technologies involved in building a Web interface to a 
massive, integrated database structure has overwhelmed previous LMDSS development 
teams. Additional funding to support developer training is one solution to dealing with 
rapidly changing technology. A second solution is to outsource using skilled contractors 
with a proven track record of successfully completing such large-scale database projects. 


Either alternative will require additional funding commitments by NAVAIR. 


i) Lack of User Input 


The literature review showed how the lack of user input and feedback is a 
common problem among failed software development projects. Strategies implementing 
user focus groups, user meetings, and periodic e-mail newsletters are all effective means 
of correcting this problem. Getting back in touch with users should begin with an 
| ageressive attempt to update the entire LMDSS user database maintained by NAVAIR. 
Multiple attempts by researchers to reach a random sample of LMDSS users by phone 


and/or e-mail resulted in successfully contacting 74 percent of the sample. The vast 
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majority of the sampled users had changed e-mail addresses since registering with 
NAVAIR. Correcting the e-mail addresses of the remaining LMDSS users will require a 
dedicated effort but will be worth the effort if a LMDSS e-mail newsletter is ever 


implemented. 


2) Personnel Turnover 


LMDSS has had more than its share of personnel turnovers with respect to 
both program managers and programmers over its eight-year history. Interview findings 
identified funding cuts as a major cause of the downsizing and personnel turnovers 
suffered by the LMDSS program. Three NAVAIR managers interviewed stated these 
turnovers had a negative effect on the LMDSS project. These frequent turnovers also 
made it difficult to document a detailed history of the LMDSS project. 


2. Does LMDSS meet user requirements? 


During the majority of this research period, LMDSS was involved in a major 
overhaul. The survey responses and interviews confirmed that before LMDSS was taken 
offline in December 1998, it did not provide timely, accurate data to its users. 

Developers have considered LMDSS to be in an evolutionary, prototype stage for 
several years with since its transition to the Web in 1994. LMDSS users were not clearly 
informed of its prototype status and were not notified of the data error/corruption 
problems. Moore and Snyder (1998) also identified this lack of communication between 
NAVAIR and LMDSS users during their research. 

The limited LMDSS data loads (which included only a few of the Navy's many 
aircraft type, models, and series) combined with frequent errors within the data itself 
both confused and frustrated users. The majority of the random sample of queried 
LMDSS users abandoned the application after a few failed attempts to acquire useful 


data. Despite this, user interviews revealed a common eagerness to know the status of 
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LMDSS, if it was finally working, and, if so, whom they needed to contact to get access 
again. 

Until LMDSS can provide access to timely, accurate AV-3M data relating to the 
Navy’s entire aircraft inventory, LMDSS will continue to be of little value to its users. In 
addition to the conversion of all AV-3M data onto the new Y2K compliant server, 
developers are currently struggling to implement a data warehouse structure where all 
aviation maintenance and logistics databases are integrated, compatible, and easily 


accessible through LMDSS and other NALDA II applications. 


3. What are the significant decisions users would like LMDSS to 

support? 

The vast majority of current LMDSS users are data analysts assigned to the 
various organizations listed in table 7. Survey responses and interviews revealed LMDSS 
is used more as a database interface and report generator than as a decision support tool. 
Questions in the Web survey designed to capture the significant decisions LMDSS needs 
to support were difficult for users to answer. Frequently, users answered with requests 
for specific types of data instead of addressing the types of decisions such data need to 
support. This was particularly apparent in the survey findings from questions 18 and 37. 


Interviews revealed major decisions supported by LMDSS users involve: 


6 Life-cycle costs 

e Reliability centered maintenance decisions 

e Historical performance of aircraft systems, parts and engines 

® The impact of program decisions on readiness 

e Comparing the performance of maintenance activities at all levels 
@ Forecasts of the future reliability of aircraft systems 
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PMAs and APMLs, who for the most part are not direct users of LMDSS, need to 
be queried more thoroughly to gain sorely needed insight into what specific decisions 


need to be supported on a continual basis. 


4. What are the key attitudes and opinions of the LMDSS users? 


The statistical ANOVA testing conducted on the four group comparison questions 
described in methodology, did not result in a statistically significant difference among the 


various user groups with respect to: 


e User requirements 
® Modeling and simulation 
e Graphics 


e Exporting LMDSS data 
e Ad-hoc query tool 


® How often LMDSS is accessed 


Some of the key attitudes expressed by LMDSS users during this research were: 
e Users placed a strong emphasis on receiving accurate and timely data. 


e Users have been frustrated with the limited data and quality of data 
available through LMDSS. Users have abandoned LMDSS until it 
delivers the data they need. 


e Users want to be able to conduct their own ad-hoc queries within LMDSS. 
Most feel that additional IQ training is required. 


e Users want to be able to export data downloaded from LMDSS into a 
familiar user tool like Excel so that they can do graphical presentations. 
An internal graphical capability would be welcomed but the primary 
concerns were exportation of data and ease of use. 
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® Based upon their past experiences, users do not trust LMDSS as a reliable 
source of data and therefore want to know how data is derived and 
defined. 


® Users overwhelmingly agreed that e-mail is the best way for NAVAIR to 
stay in touch with its customers at all levels. 


5. How has LMDSS been funded? 


In the past, LMDSS has received its funds from the NALDA program funding and 
has never been assigned its own line of funding. LMDSS managers identified severe cuts 
in funding and personnel as the major causes for many of the delays and problems 
suffered during the eight-year LMDSS development effort. Recently, much of the 
funding for LMDSS development efforts has come "out of hide" from NAVAIR's 
Operation and Maintenance funds. 

As NALDA I is closed down in October 1999, LMDSS will take on a far greater 
importance as the primary gateway to all of the Navy's AV-3M historical data. An 
ageressive POM effort needs to be, initiated to acquire the funding necessary to finish 
existing plans for LMDSS and to ensure proper funding for maintenance and future 
development 


6. How can LMDSS be improved to better meet managerial needs? 


a) Provide accurate and timely data. 


A common perception held by LMDSS users is that most AV-3M data 
errors occur at the hangar deck level because maintenance technicians do not have an 
appreciation of how important it is to provide correct maintenance and logistics data. An 
effort to work with the Chief of Naval Education and Training (CNET) to include more 
detailed NALDA/LMDSS training in the appropriate A schools and technical training 
centers would go far to help correct this data entry problem. A CD-ROM multimedia 
tutorial mailed to all Fleet aviation commands might be one way to alleviate this 


problem. Such a tutorial could show maintenance and logistics technicians how 
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important AV-3M data is to making correct strategic program decisions at the PMA 
level. 

An aggressive effort by NAVAIR is needed to hold wings and squadrons 
accountable for the quality of AV-3M data submitted. Linking this effort to periodic 
command inspections would do much to motivate squadrons to pay closer attention to the 


quality of their submitted data. 


b) Make LMDSS more user friendly. 


Although a monumental effort has been made to make LMDSS easier to 
use, only twenty-six percent of the users who responded to the Web survey considered 
LMDSS to be user-friendly. The poor quality of LMDSS data and the significant effort 
involved in accessing needed data surely contributed to this attitude. 

An idea promoted by the current NAVAIR 3.6.2 would be to customize 
LMDSS for each individual user in a manner similar to the Yahoo and CNN Web sites. 
Upon first registering at the LMDSS site, a user could customize the site to provide only 
specifically required data and reports. This personalized site environment would appear 
every time the user subsequently logged-on to LMDSS. All desired data and reports 
would be updated and available every time the user visited his uniquely designed LMDSS 
site. Automated tools could be used to inform the user when requested metrics of interest 
had changed. Current Web technology that pushes data to personalized Web sites could 
be implemented to continually update a user's individualized "My LMDSS Home Page" 
‘with the most current data available. Personalized icons could be created by the user on 
this home page to quickly bring up commonly needed reports containing the most recent 
data. 

A file folder tab format as found on many of today's popular Web sites 
might help LMDSS users find their way to desired data easier than requiring them to drill 
down into imbedded modules. Interviews revealed users want more extensive help 


screens and help functions. Users place a higher importance on just getting accurate, 
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timely data than details concerning report formats. As one analyst stated, "Using LMDSS 


needs to be as easy as following the bouncing ball." 


Cc) Enable users to easily export LMDSS data to an application 
supporting graphics and spreadsheets like MS Excel. 


LMDSS users strongly expressed their need to cleanly export data to 
applications with graphics capabilities like Excel. The survey questions relating to this 
topic received the most responses in both quantity and in the strength of the users' 
conviction. (See results of questions 24 and 32 in the Results section of Findings.) 
Although internal graphical capabilities were also desired, the users were clear that they 
needed to be able to cleanly export data to complete further analysis on the material 
provided by LMDSS. Users interviewed seemed willing to make compromises on 
specific data format as long as the data was accurate, timely, and could be downloaded to 


an application and manipulated by the user. 


d) Revise the IQ tool to be more user friendly or abandon the 
current IQ tool and develop a new, easier to use, ad-hoc query 
_ tool within LMDSS. 


LMDSS users place a high importance on their ability to develop 
individualized ad-hoc queries. Users familiar with the IQ query tool within LMDSS 
reported it as too difficult to use. Only two percent of LMDSS users surveyed found the 
IQ tool to be user friendly. Users expressed their frustration upon completing IQ tool 
training when they discovered they were unable to overcome LMDSS access and 
software difficulties. Others reported losing most of the IQ skills acquired during 
training when they were unable to practice using the tool because of various problems 
accessing LMDSS data. 

; A search engine similar to those found in other Web sites (altavista. com, 
dogpile.com, yahoo.com, amazon.com) is one suggestion for making ad-hoc queries 
easier to implement. Data contained in common report templates could be developed 


with the help of extensive user interviews. These reports could be easily accessed by an 
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Internet style search engine interface familiar to most Web surfers. Such a search engine 
would make it easier for users to conduct a customized search. The search engine would 
hide the complicated details of SQL coding from the user just as Web site authoring tools 
now hide the complex details of HTML code. 


e) Provide a modeling/simulation capability. 


If LMDSS is ever upgraded to become a fully functioning decision support 
tool, as its name implies, jt will have to eventually include extensive modeling and 
simulation capabilities. As shown in the literature review, modeling and simulations are 
an essential part of a DSS. Previous LMDSS research by Moore and Snyder (1998) 
determined modeling is needed by both PMAs and APMLs to better forecast life-cycle 
costs and predict the effect of program decisions on those costs and overall readiness. 

The active involvement of PMA/APMLs will be required to develop the 
models and simulations needed to meet their unique decision support needs. DSS 
modeling requirements will have to be thoroughly documented and financially supported 
in the Program Objectives Memorandum process by NAVAIR senior leadership to be 
successfully implemented. | | 


Some of the modeling applications LMDSS users suggested included 


models that would: 
® Assist in predicting future maintenance support requirements 
@ Estimate life cycle costs and the impacts of aging 
e Capture various return on investments for proposed readiness and program 
changes 
° Forecast spare costs and failure trends 


A prototype model built by the authors of this thesis provides an example of how 
modeling and simulation could assist logistics and maintenance managers. The model, 


designed to help wing maintenance managers forecast the readiness impact of a real- 
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world depot support problem, is presented in Appendix F. The model enables users to 
conduct sensitivity and "what if" analysis to assist in making strategic, unstructured 


decisions. 


Vs How can NAVAIR improve the LMDSS program? 


NAVAIR has several major difficulties facing them as they struggle to make 
LMDSS an effective analysis tool for the Fleet. These include: 


® Ensuring the final LMDSS product performs properly within established 


quality specifications. 

e Ensuring LMDSS meets user requirements. 

® Convincing users LMDSS is capable of providing accurate, meaningful 
data. 

® Training new and repeat users how best to use LMDSS. 

@ Continuing to develop new and/or better capabilities into the system. 


Although the findings of this research discovered numerous areas that could be 
improved upon, the following are the significant areas of concern identified by the 


research: 


a) Improve communications with users 


Managers interviewed within the LMDSS development team believed the 
‘experts on the QA team and others at NAVAIR knew what the LMDSS users wanted 
without asking the users themselves. NALDA user meetings were abandoned as NALDA 
Il and LMDSS underwent development with the intention of reinstating them in the 
summer of 1998. Months turned into years without a NALDA or LMDSS user meeting. 
Over eighty percent of the users participating in the Web survey reported they had not 
been contacted for input by the LMDSS development team. Only three percent reported 
attending NAVAIR sponsored LMDSS user group meetings. Given the project’s history 


of missing deadlines and not meeting advertised expectations, a dedicated effort needs to 
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be made to improve communications between the NAVAIR developers and the Fleet 
customers. This includes identifying a timely means to provide information to the in 
on the status of LMDSS and a means of obtaining feedback from the users. Elimination 
of miscommunication will go a long way toward reestablishing a favorable reputation of 
LMDSS within the user ranks. 

Teleconferencing, focus groups, and regional user meetings are alternative 
means to continuing the critical effort of staying in touch with LMDSS customers in a 
fiscally constrained environment. Accurate user feedback and input is critical to 
maintaining meaningful LMDSS software requirements. Assuming those on the LMDSS 
development team already know what LMDSS users want and need is a risky 
assumption. 

Survey respondents suggested NAVAIR stay 1n touch with users through . 
the use of an email LMDSS newsletter. Since 99 percent of the users within the LMDSS 
random sample contacted during this research have email accounts one can assume the 
majority of the total LMDSS user population also have email accounts. Only two users 
contacted were aware of the data problems within LMDSS that resulted in data being 
corrupted and frozen for several months in the fall of 1998. During this period, users 
were still going to the LMDSS Web site for periodic data searches. A NAVAIR 
sponsored LMDSS e-mail newsletter would help avoid such misunderstandings, maintain 


the trust and cooperation of users, and assist greatly in collecting user feedback. 


b) Improve LMDSS training. 


User comments emphasized the importance of hands-on training with 
students being assigned to their own computer terminal and practicing with a more 
realistic sample database. Users expressed their frustration of completing LMDSS 
training only to discover LMDSS was still under development, delivered inaccurate data, 
and did not work as promised. 

To avoid these misunderstandings, further training should be delayed until 


LMDSS is fully operational. The abundance of non-response answers to survey questions 
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and the interview results reveal users have abandoned LMDSS and will continue to do so 
until it is fixed. Continuing to conduct training before LMDSS can deliver as promised 
will only worsen NAVAIR's relationship with its customers. 

CNET's Electronic Schoolhouse Network (CESN) should be explored to 
expand the scope of LMDSS training via distance learning. CNET has built several 
electronic classrooms that are available to support LMDSS training. All of the CESN 
sites are located at the major Fleet concentration areas. In 1997, a CNET pilot project 
was conducted to test the viability of completing hands on software training via distance 
leaning. Software training originating at the Naval Reserve Professional Development 
Center in New Orleans was transmitted to remotely located electronic classrooms 
operated by CESN. Software interface difficulties limited the success of this pilot 
project, but CNET has continued to work toward improving the viability of distance 
learning and is working on attaining the capability for CESN to conduct remote software 
training. Utilizing CESN sites would also allow NAVAIR to conduct timely, periodic 
LMDSS focus groups and user feedback meetings while avoiding cost prohibitive travel 


expenses. 


112 








Vil. CONCLUSIONS/RECOMMENDATIONS 


A. THE NEED FOR A STRATEGIC DECISION SUPPORT SYSTEM 


1. Conclusion: LMDSS does not meet the original requirements 
established for a strategic decision support system for PMAs/APMLs. 


The original requirement for LMDSS was established by NAVAIR 04 and the 
former Aviation Supply Office (currently renamed NAVICP) in June 1991. This 
requirement directed the development of a strategic decision support system to assist 
PMA, APML, and WSM teams in measurably reducing program life cycle costs while 
minimizing negative effects on F leet readiness. Many specific DSS capabilities were 
initially promised to the NAVAIR chain of command in 1993. It was believed the 
developed program LMDSS would: 


e Provide a single source of real-time maintenance and deeucs 2 data by 
accessing as many as nine stovepipe databases 


® Allow sensitivity or “what if’ analysis based on the ability to manipulate 
supply and maintenance parameters 


® Support detailed causative research 
® Provide an audit trail of a manager's decision process 
e Forecast engine problems and repair alternatives using the corporate 


expert knowledge of top engine analysts 


e Assess the repair productivity and efficiency of maintenance activities 
across all organizational levels 


e Deliver presentation quality graphic (Naval Air Systems Command, 
LMDSS Requirements Document, 1993) 


The current operational version of LMDSS has none of these capabilities. During 
its eight-year development, LMDSS has evolved from a DSS design for PMs and APMLs 


into a limited management information system (MIS) used by data analysts at all levels. 
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Recommendation: Rename LMDSS and call it a management 
information system. Validate the requirement for a strategic decision 
support system to help PMAs, APMLs, and WSMs better manage 
their programs. 


LMDSS users contacted during this research expressed their loss of faith in 
LMDSS' capabilities as the primary reason for abandoning it. Moore and Snyder (1998) 
concluded LMDSS did not meet the criteria for a DSS. Consideration should be given to 
changing the name of LMDSS to more accurately reflect its role as a Web interface that 
provides AV-3M data for Fleet and civilian data analysts. The capability of LMDSS to 
drill down into AV-3M data, identify primary cost drivers, present standardized reports, 
and support ad-hoc queries places it in the management information system category. 
When LMDSS becomes fully operational a new name might help restore user faith and 
avoid misunderstandings. A more accurate name for LMDSS would be the Logistics 
Management Information System (LMIS). . 

The original requirement for.a strategic decision support system to improve 
program management needs to be further explored and validated again. If this 
requirement is determined to still be valid, we recommend a strategic decision support 
system containing models and simulations and supported by an integrated data 
warehouse be built for NAVAIR program managers. Such a system could provide 
invaluable assistance to PMAs, APMLs, and WSMs in their daily struggle with 


unstructured decisions to overcome limited funding and maintain readiness. 


me Conclusion: The key to providing accurate, timely logistics and 
management data lies in the development of a data warehouse. 


An effective strategic decision support system for NAVAIR managers will require 
the integration of data currently found in stovepipe, legacy and relational databases 
within NALDA I. A three-tier, data warehouse architecture, containing a meta-data 
capability, with a middleware based, application interface holds the best promise for 


achieving this integration. 
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Recommendation: Develop a data warehouse to support LMDSS. 


Developing a data warehouse that seamlessly integrates numerous NAVAIR 
legacy databases to provide easy access to timely, accurate data is a major undertaking 
for any organization. A detailed, thorough effort to benchmark other organizations that 
have successfully completed such a project should yield invaluable "lessons-learned." 

To build a data warehouse, transformation and extraction tools will be needed to 
access the various data types. Meta-data managers will be needed to describe and track 
the origins of data, how it was cleaned and distributed, who 1s responsible for it, and how 
frequently it is updated. In addition, business intelligence tools and data cleansing tools 
will also be required. (Malloy, 1997) 


B. SOFTWARE DEVELOPMENT ISSUES . 


1. Conclusion: The absence of a formal risk management program Is a 
major contributor to the failure of the eight-year LMDSS program. 


A properly implemented, formal risk management program is a proven software 
development tool for identifying, evaluating and mitigating potential risks. The literature 
review chapter describes research showing the impact an effective risk management 
program can have on the success of a software development effort.. Although interviews 
led us to believe formal risk management practices are used in other areas of the NALDA 
program, the LMDSS project does not appear to have a risk management policy in effect. 

Recommendation: Institute a formal LMDSS risk management 
program 

Formal policies requiring periodic reviews of existing and emerging risks present 
a proven method for software managers to manage the potential risks to a software 
development project. Risk management will allow the LMDSS development team to 
anticipate obstacles and proactively develop strategies to mitigate them. 

Although a variety of formal methods exist, developing and tracking an up to date 
list of "Top Ten Risks" to the LMDSS program is a recommended, proven strategy of 


communicating potential obstacles to both managers and developers (McConnell, 1996). 
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As risks are controlled and minimized they can be removed from the list and replaced by 
higher priority risks. 


2. Conclusion: Documentation in all areas of the LMDSS program needs 
to be improved. 


As stated in the discussion section, no evidence could be found suggesting the 
LMDSS project used automated software tools, CASE tools or standardized software 
metrics to develop accurate estimates for planning and workload scheduling. Without 
accurate, credible estimates of man-hours required, anticipated lines of code and other 
proven software development metrics, schedules and deadlines became best guesses. 
The inability to meet poorly estimated deadlines put additional unneeded pressure on 
LMDSS team members. Poor documentation also made it difficult for LMDSS program 
managers to fight for limited funds. As a result, the available funding drove LMDSS 
requirements instead of the requirements driving the level of funding. This resulted in a 
“death spiral” causing further delays and decreased capabilities. 


Recommendation: Improve the documentation maintained by the 
LMDSS software development team. 


In the same way that trend analysis and configuration management are 
instrumental in improving aviation maintenance practices, documentation is just as 
important to software development. Metrics need to be identified and tracked in order to 
measure progress (or lack of progress) in the LMDSS program. Proper metrics are 
imperative in order to implement any quality assurance programs. Quality cannot be 
improved is there are not any standardized methods with which to identify performance 
or process problems. Risk management documentation and meticulous, thorough 
schedules based on accurate estimates from software metrics can be influential tools in 
future attempts to attain additional LMDSS funding support from upper management. 

CASE tools should be reviewed for applicability and implemented where 
appropriate. A dedicated effort needs to be made to document program metrics such as 
man-hours expended, Lines of Code (LOC) completed, and software errors per LOC (bug 


density). Contractors should also be providing similar metric data on their development 
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process for analysis. This data should provide managers with a better means on how best 
to allocate funding and manpower resources to meet cost and time constraints. Data 
sampling is another method for verifying accuracy. In addition, accurate documentation 
is crucial to conducting thorough post-mortem (or lessons learned) summaries to prevent 
repetition of process problems in future software projects. 


3 Conclusion: Major LMDSS requirement changes significantly 
contributed to its delayed deployment. 


LMDSS managers continue to propose major requirement changes prior to 
successfully completing the current phase of development. As shown consistently in the 
software development literature, the delays and costs associated with changing core 
requirements late in the developmental cycle is exponentially proportional to the length 
of time in the cycle. More simply stated, the earlier a change is implemented, the less 
impact it will have on schedule and cost. When changes are made in the middle or late in 
a SW development project, it does not matter whether the reason is due to changes in 
funding, technology and personnel. The result will be the same: drastic schedule delays 
and/or exorbitant cost increases. LMDSS suffered every possible requirement change 
including hardware requirements, software language, contractor changes, personnel 
changes and user requirement changes. Given this, it is no wonder that LMDSS is over 
budget, behind schedule and not fully functional. 


Recommendation: Identify and validate detailed requirements for 
LMDSS and implement a versioning strategy to manage future 
requirements changes. 


LMDSS has gone through so many metamorphic changes users do not have a 
clear understanding of its current capabilities. Program managers need to get back in 
touch with LMDSS users and thoroughly document their requirements. When LMDSS is 
fully deployed, NAVAIR needs to properly advertise its strengths and limitations to its 
customers. As shown in the findings, the majority of users did not know specifically 


what LMDSS could do. 
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Implementing a versioning technique, similar to Microsoft’s tactic of developing 
its MS Office line, would enable continued developmental improvements while still 
focusing programmers on meeting core program requirements. LMDSS needs to adopt 
a policy that mirrors Microsoft’s Office marketing practices. Program managers need to 
identify a core set of capabilities that LMDSS should have and strive to complete 
production. Major changes resulting from new technologies, new personnel or funding 
constraints should be reviewed for potential integration into LMDSS. [If evaluations of 
new requirements are favorable, a plan should be developed to implement the new 
requirements into a future version of the application instead of trying to continually 
change the existing development process. 


4, Conclusion: Lack of continued user involvement in the LMDSS 
development project contributed to its failure to meet customer needs. 


As covered extensively in the Findings chapter and again in the Discussion and 
Analysis chapter, communications between users and developers is poor. Over eighty 
percent of the users participating in the Web survey reported they had not been contacted 
for input by the LMDSS development team and only three percent reported attending 
NAVAIR sponsored LMDSS user group meetings. Due to the years of conflicting reports 
about the capabilities of the LMDSS application, most users do not have a clear 
understanding of what exactly the LMDSS program can do. As evidenced in the survey 
results, many users are willing to provide input but have not been asked. 


Recommendation: NAVAIR needs to improve communications with 
LMDSS users. 


Findings showed users prefer a NAVAIR sponsored LMDSS e-mail newsletter 
but teleconferencing, focus groups and regional user meetings are alternative means of 
improving user feedback and ensuring two way communications. 

In order for LMDSS to better meet user requirements, NAVAIR managers need to 
actively involve users in the development process. Periodic LMDSS user meetings, focus 
groups, and aggressive e-mail correspondence are recommended strategies for NAVAIR 


to employ to get back in touch with its LMDSS customers. 
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5. Conclusion: Until LMDSS can provide access to all AV-3M data for 
every platform in the Navy’s current inventory, LMDSS will continue 
to not meet user needs. 


As stated before, without AV-3M data properly loaded for all type/model/series 
and type equipment codes, LMDSS will continue to provide only limited usefulness. 
Interviews with NAVAIR managers revealed LMDSS developers failed to appreciate the 
complexity of building an integrated database support structure. Y2K compliance 
measures, inadequate contracting and data security concerns further complicated the 
transition of data into the system. This has resulted in repeated difficulties loading the 
data. 


Recommendation: Complete the loading of AV-3M data for all 
type/model/ series aircraft to enable access by LMDSS users. 


All data needs to be loaded as soon as possible so that all users may obtain the 
data required. Future LMDSS development efforts need to be supported by adequate 
funding, skilled data warehouse experts, and a major contractor with a track record of 


implementing similar data warehouse projects. 


C. AREAS FOR FURTHER RESEARCH 


The following areas for future LMDSS research are: 


° Conduct a follow-on survey of LMDSS users once LMDSS is fully 
functioning with access to all historical aircraft and logistic data. 


@ Explore the NALDA II data warehouse plan and research alternative 
strategies to support LMDSS integration. 


® Conduct an in-depth analysis into the causes of LMDSS data errors and 
recommend ways to improve the quality of LMDSS data reports. 


® Validate and define the need for a strategic DSS by NAVAIR program 
managers. 


® Develop models to assist NAVAIR program managers in making strategic, 
unstructured decisions with the goal of improving readiness while cutting 
program costs. 
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APPENDIX A. CLASSIC SOFTWARE DEVELOPMENT MISTAKES 


McConnell (1996) identified and described thirty-six classic mistakes that directly 
lead to a failed software development effort. His list is divided into the four categories of 


people, process, product, and technology. 


1. People 


e Undermined motivation 

e Weak personnel 

e Uncontrolled problem employees 

e §=©Heroics 

e Adding people to a late project 

e Noisy, crowded offices 

e Friction between developers and customers 
e Unrealistic expectations 

e Lack of effective project sponsorship 
e Lack of stakeholder buy-in 

® Lack of user input 

e Politics placed over substance 


e Wishful thinking 


2. Process 


e Overly optimistic schedules 


12] 








Insufficient risk management - 

Contractor failure 

Insufficient planning 

Abandonment of planning under pressure 
Wasted time during the fuzzy front end 
Cuts to requirements analysis, architecture, and design 
Inadequate design 

Shortchanged quality assurance 
Insufficient management controls 
Premature or overly frequent convergence 
Omitting necessary tasks from estimates 
Planning to catch up later 


Code-like-hell programming 


3. Product 


Requirements gold-platting (more requirements than needed) 
Feature creep (changing requirements) | 

Developer gold-platting 

Approval of a schedule slip while adding additional requirements 


Research-oriented development 


4. Technology 


Silver-bullet syndrome (a single new practice will solve schedule problems) 
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e Overestimated savings from new tools or methods 
e Switching tools in the middle of a project 


e Lack of automated source-code control 
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APPENDIX B. NALDA Il 


The Naval Aviation Logistics Data and Analysis (NALDA) system is an 
automated, relational database and information retrieval system for aviation logistics 
management and technical decision support. NALDA Phase I first became operational in 
the early 1980s and evolved from the need to improve the data analysis capabilities of 
logistics and aviation program managers as Navy's aviation weapons and support systems 
became more complex and sophisticated. In the mid 1990s, NALDA Phase I expanded 
the data base and analysis capability to include all Fleet logistics elements. (SPAWAR, 
1999) 

The primary objectives of the NALDA Phase II system is to leverage information 


technology to achieve process improvements through the following: 


° | More timely and accurate information and tools . 

® Affordable readiness and total ownership cost metrics and process tools for 
decision makers 

® Easier information access to managers, logisticians, engineers and maintenance 


technicians at all levels within the Fleet 
@ Create and store data once, and use it many times 


@ Integrate with the Department of Defense's Standard and open systems 
architecture, minimizing stovepipes, redundancies, and islands of automation. 
(NAVAIR, 1999) 


All NALDA II applications, one of which is LMDSS, will be structured within a 
single shared, logical, and physically distributed Integrated Weapons Systems Data Base 
(IWSDB). The IWSDB will be a centralized Oracle object-oriented data base structure 
_ that will contain all relevant naval aviation logistic data. (Joseph Interview, 1998) 
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APPENDIX C. FOCUS GROUP QUESTIONS 


Initial questions: 
Q Who are the Users? List Categories/Activities 
What are the levels of users? 
Demographics? Provide information about the users. See below. 
What are the users doing with LMDSS now from your perspective? 
What are the most common help requests? 
What do the users want to do with it now? 
What is the future of LMDSSS going to look like? 
What do you like/dislike about the old survey? 


Oooodododao oO 


Additional questions that arose during the focus group 

Do TYCOMS hire people that may also have/need access? 

Do contractors have restricted access? 

What information do you want to receive from the survey? 
What about the change from Unix to the Web? 

Do you think the lost graphical capabilities will be a problem? 


Oo oO 


oO 


Demographics recommended by focus group: 
OQ Command/Activity 
QO Type Users 
Q Active Duty 
oO Contractor 
ag Civilian 
oO GS Worker 
Job Title 
IT Experience 
Rate/Rank 
Internet access 
O Frequency 
QO Type connection ~- modem/network connection 
Q Type organizational category 
o AIMD 
Q APML 
oO Staff 
Q Depot 


Ood oO 
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APPENDIX D. WEB-BASED SURVEY 


Logistics Management Decision Support System (LMDSS) 
Survey 


Thank you for your assistance in this LMDSS research effort sponsored by NAVAIR 
3.6.2. This survey was prepared by CDR Mark Krause (mwkrause@mbay.net) and 
LCDR Ellen Evanoff (evanoffrus@aol.com) from the Naval Postgraduate School. The 
purpose of our research is to assist NAVAIR with the identification of LMDSS user 
requirements in order to develop a better decision-making tool for the Logistics and 
Maintenance communities. The quality of the data from this survey will depend solely 
on the honesty and accuracy of your input. 


In Feb 99, LMDSS was removed from the Web and is currently being reworked and 
updated. Despite the fact LMDSS will not be available for use for several months, we 
are very much interested in your past experience with LMDSS and what you would like it 
to do for you in the future. | 


The identity of those responding to this survey will be kept strictly confidential. Some 
basic information is requested in case we need to contact you at a later date. If there are 
any questions, both CDR Krause and LCDR Evanoff can be reached via their email 
addresses listed above. So, lets get started! 


Q.Name 
What is your name? 


Q. Email 
What is an email address we may use to contact you? 


Q.Command 
Please provide your command name (or name of business) and job title. 
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1. How often did you access LMDSS when it was available on the Web? 
Infrequently 

Monthly 

Weekly 

Daily 

Don't know 

Not applicable 


QVOO0O00 


OSOCOOOCCOE COU0CCOCOOOCO OO 


QQOQO0Q0000 


_ Choose the term below which best describes your organization: 


Fleet Command (CINCLANT,CINCPAC,COMTHIRDELT...) 
Type Command (COMNAVAIRPAC,COMNAVAIRLANT.,..) 
Systems Command ponenenereen meee: SPAWAR...) 
COMNAVRESFOR 

COMNA VAIRESFOR 

Naval Safety Center 

Marine Corps Activity 

Navy Inventory Control Point 

Depot 

AIMD 

WING 

Squadron 

Civilian Firm under contract to the Navy 

Other 


ow long has it been since you last accessed LMDSS? 


Less than 6 months 
6 -12 months 

1] - 2 years 

2 - 3 years 

3 - 4 years 

4-5 years 

More than 5 years 
Don't remember 
Not applicable 


. When were you first introduced to LMDSS? 


Fewer than 6 months ago 
6 -12 months ago 

| - 2 years ago 

2-3 years ago 

3 - 4 years ago 

4-5 years ago 

More than 5 years ago 
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OQ Don't remember 
O Not applicable 


5. When the LMDSS Web site comes back online, what will be your primary means 
of accessing it? 
28K modem 

56K modem 
ISDN connection 
T-1 connection 
Via the network 
Don't know 
Other 


QOOQO0000 


at kind of problems did you have accessing LMDSS? (Check all that apply) 
I have not had any problems. 
Firewall blocking access 
Proxy difficulties 
Downloading matrix tables 
Unable to access the IQ tool 
Computer hardware inadequate 
Internet browser issues 
Other 
Don't know, just couldn't access the LMDSS web site 


6. 


pocoooocooos 


7. When LMDSS is back online, users will be required to use Netscape 
Communicator 4.5 with 128 bit encryption (available for free download at 
www.netscape.com). Do you anticipate any problems downloading and installing 
this browser? 

O Yes 

O No 

O Don't know 

O Not applicable | 


8. What kind of software problems did you experience while using LMDSS? (Write 
NA or "No problems", if appropriate) 


9. What kind of information do you expect to retrieve from LMDSS when it 
becomes available? (Write "Don't Know" or NA, if appropriate) 


131 








10. What information do you require, but are unable to access via LMDSS? (Write 
"Don't Know" or NA, if appropriate) 








11. How difficult was it to build matrix tables within LMDSS? 
1 Impossible 

2 

3 

4 

5 Easy 

Not applicable 


QOWQQWOO0OO 


12. Which LMDSS modules did you access the most (check all that apply)? 
Management Analysis 
Candidate Identification 

Trend Analysis 

Cost Analysis 

Detailed Analysis 

Supply Analysis 

Engine Analysis 

Reference Information 
Application Management Tools 
Change Requests 

Feature Synopsis 

Other 

Don't know 

Not applicable 


OO0OOCOCOCCOCOCO 


13. Have you ever used the IQ tool to build any custom queries? 
OQ Yes 
OQ No 
QO Don't know 
O Not applicable 


14. Do you think the IQ tool was user friendly? 
OQ Yes 
OQ No 
O Don't know 
O Not applicable 


15. Do you think LMDSS was user friendly? 
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OQ Yes 
QO No 
OQ Don't know 


16. How important do you consider the addition of a structured, ad-hoc query tool to 
LMDSS? 
Extremely important 
Very important 
Somewhat important 
Not very important 
Not at all important 
Don'tknow _ 
Not applicable 


VO0O0000 


17. If you know of other NALDA applications which contain query formats you 
would like to see implemented in LMDSS please list them below? (Write "Don't 
know" or NA, if appropriate) 








18. Please list a few of the key decisions you would like LMDSS to support when it 
comes back online. (Write "Don't Know" or NA, if appropriate) 











19. LMDSS will collect much of its raw data from NALCOMIS. If there is any data 
you will require which is not captured by NALCOMIS, please describe it below? 
(Write "Don't know" or NA, if appropriate) 











20. How important do you consider the development of a modeling/simulation 
capability in future versions of LMDSS? 

Extremely important 

Very important 

Somewhat important 

Not very important 

Not at all important 

Don't know 

Not applicable 


VQO00000 
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21. Can you provide an example of how a modeling/simulation capability to conduct 
sensitivity (what if) analysis will be useful? (Write "NO" or NA, if appropriate) 








22. How important is it for you to know the details concerning how LMDSS reports 
are derived before you feel you can trust the information they provide? 

Extremely important 

Very important 

Somewhat important 

Not very important 

Not at all important 

Don't know 

Not applicable 


QOWOV0000 


23. How important is a detailed description or definition (data dictionary) of all 
LMDSS data? 

Extremely important 

Very important 

Somewhat important 

Not very important 

Not at all important 

Don't know 

Not applicable 


QVOQO0000 


24. How important do you consider the development of a graphics capability in 
future versions of LMDSS? 

Extremely important 

Very important 

Somewhat important 

Not very important 

Not at all important 

Don't know 

Not applicable 


QOQOQO0000 


25. Have you found the usage of definitions and metrics by LMDSS to be in 
compliance with established 3M definitions? 

QO Yes 

QO No 

QO Don't know 
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26. How would you rate the quality of the LMDSS instruction you have received 
from NAVAIR personnel? 

Did not receive any training 

Excellent 

Good 

Fair 

Poor 

Don't remember 

Not applicable 

27. Did the training include Structured Query Language training using the IQ tool? 
Yes | 

No 

O Don't remember 

OQ Not applicable 


OOY OO00000 


28. Do you feel you will need LMDSS refresher training when LMDSS comes back 
online? 

O Yes 

O No 

OQ Don't know 


29. How important is "hands on" LMDSS training with each student assigned to a 
computer terminal? 

Extremely important 

Very important 

Somewhat important 

Not very important 

Not at all important 

Don't know 

Not applicable 


QOQO00000 


30. How can LMDSS training be improved? (Write "Don't Know" or NA, if 
appropriate) 











31. Please describe any examples of your not being satisfied with the quality of the 
data you have obtained through LMDSS? (Write "None" or NA if appropriate) 
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32. How important is it for future versions of LMDSS to be able to cleanly export 
data to applications with graphics capabilities such as Excel? 


QOOQO0000 


Extremely important 
Very important 
Somewhat important 
Not very important 
Not at all important 
Don't know 

Not applicable 


33. Has anyone from NAVAIR or the LMDSS development team ever contacted you 
requesting your input/feedback concerning LMDSS? 


O 
O 
O 


Yes 
No 
Don't know 


34. Have you ever attended a NAVAIR sponsored LMDSS users’ group meeting? 


O 
O 
O 
O 


Yes 

No 

Don't remember 
Not Applicable 


35. What might some of your reasons be for using LMDSS when it returns to the 
Web? (Check all that apply) 


OUOCOOCHOOOQOOOCOOCCOC: 


Find data to support logistics acquisition decisions 

Reduce life cycle support costs of aviation systems 

Conduct trend analysis 

Search for data to complete periodic reports and forms 

Identify high cost drivers to conduct cost analysis 

Track the results of implemented improvements 

Measure the impact of decisions or policies on platform readiness 
Measure the effect of improvement actions on support costs 
Identify system degraders 

View cost data and flight hour history of aircraft engines 

Drill down to the MAF level to determine cause of system degraders 
Compare the performance of my command with similar commands 
Reliability information 

Other (Please describe in question # 43) 


36. If you selected "other" in the previous question, please describe below: 
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37. Referring to your answers in question # 35 and # 36, what do you anticipate your 
primary reason for using LMDSS will be? 








38. In what ways could NAVAIR make analysts more aware of LMDSS? (Write 
"Don't Know" or NA if appropriate) 





39. If you have any additional thoughts as to how NAVAIR could improve LMDSS 
please write them below: (Write "Don't Know" or NA if appropriate) 











137 


138 





APPENDIX E. PHONE INTERVIEW QUESTIONS 


1. What activity/command do you work for? 


What is your job title? Describe your background with IT systems. How many 
years of experience do you have with the 3M system? In what capacities have you 


worked with the 3M system? 


2. Are you an active LMDSS user? 


No: Why don’t you use LMDSS? Why do you still have an active LMDSS 
account? What would you want LMDSS to do to motivate you to use it regularly? 
Yes: How often do you use LMDSS? | 


3. | Howdo you access LMDSS? 


How do you connect to the Internet? (network, modem, ISDN, T-1) Any 
problems connecting to LMDSS via the WWW? What kind of computer do you use to 
access LMDSS? (CPU type/speed, RAM, hard drive) Any problems with firewalls? Any 
software problems related to LMDSS? What kind of browser do you use? Any problems 
downloading matrices? 


4. Have you received LMDSS training from NAVAIR? 


When was the training? Where did you receive the training? Did the 
instructors effectively teach the material? Did the training include SQL training? How 
can LMDSS training be improved? Do you feel you need refresher training. Did the 
training adequately prepare you to effectively use LMDSS? Why or why not? 


5. | Have you used the Help Desk? 


No: Why not? 
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Yes: What is your opinion of the performance of the Help Desk? How often 
do you call the Help Desk? Any problems reaching the Help Desk? Are the Help Desk 
personnel able to adequately answer your questions? What are your most common Help 


Desk requests? 


6. How useful is LMDSS? 


What questions are you usually trying to answer with LMDSS? Do you 
consider LMDSS to be user friendly? What information does LMDSS not provide? Any 
problems building the matrices? Which matrices do you access the most? Is the 
information easy to get from LMDSS or are there too many steps? Does the flow of data 


seem logical and easy to follow? 


ds Have you used the IQ tool to build any SQL queries? 

Do you feel comfortable using the IQ tool to build SQL queries? Is the IQ tool 
user friendly? Should a more structured, ad-hoc query tool be developed that’s easier to 
use? Do any other NALDA applications contain query formats you would like to see 
used in LMDSS? | | 


8. What are the key decisions you would like LMDSS to support? 


Does LMDSS provide the information you need to make these decisions? 


What data do you need which is not captured by NALCOMIS? 


9. How would a modeling capability be helpful in supporting your 
research? 


Describe modeling as models or simulations, which allow the user to test 
' “what if? scenarios. Discuss whether sensitivity analysis (changing variables to see how 
the outcome is affected) is an important capability for LMDSS users. Probe for specific 


examples or stories that show how useful modeling might be for LMDSS users. 
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10. How would a graphics capability be useful? 


Probe for specific examples or stories. Would the capability to export to an 


application with a graphics capability such as Microsoft’s Excel be useful? 
11. What is the best way NAVAIR could make analysts more aware of 
’ LMDSS? 


How did you become aware of the LMDSS and its capabilities? 


12. How would you improve LMDSS to make it better? 


Probe for examples. 


141. 
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APPENDIX F. SH-60B LOGISTICS MODEL 


As shown in the review of the literature, models provide a DSS with powerful 
analytical capabilities. Sensitivity analysis using models and simulations allows 
managers to change problem variables to measure their impact on the final outcome of a 
proposed alternative solution. Although LMDSS does not contain models or simulations, 
this appendix provides an example of how modeling can assist managers in the 
evaluation of alternatives to a real world, unstructured problem. 

The SH-60B LAMPS MKIII helicopter community has been struggling to 
overcome a readiness degrader involving a shortage of Ready for Installation (RFI) main 
rotor blade spindles. The primary cause of the shortage is the backlog of spindles 
awaiting maintenance at the H-60 Depot in Corpus Christi, Texas. The Depot is tasked 
to repair H-60 spindles submitted for overhaul by both the Army and Navy. Due to 
shortages in manpower, training, and funding the Depot has been unable to keep up with 
the demand for RFI spindles. As a result, Navy SH-60B helicopters often remain in a 
None Mission Capable (NMC) status for one to three months awaiting a RFI spindle 
from the supply system. 

The purpose of this model is to provide NAVAIR 3.6.2 with a prototype tool 
capable of sensitivity analysis and "what if" projections. The global time variable for the 
model is in hours. The time frame for each model run is 88,400 hours or approximately 
10 calendar years. Figures 18 and 19 are the two output graphs provided at the end of 
each model run. Figure 18 shows the result of a 120 hour (5 day) delay at the Depot 
given the assumption there are 336 "flying" spindles assigned to 84 helicopters in the 
Pacific Fleet (PACFLT) and 20 spare spindles in the supply system. In Figure 18 the top 
blue area of the graph shows only 304 RFI spindles in PACFLT after 10 years with the 
black line showing 52 of the total 356 spindles are in the Depot awaiting maintenance. 
The red line depicting the number of spindles in the supply system is hidden within the 


horizontal 
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Figure 19 SH-60B Spindle Readiness Plot 
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graph axis showing a steady value of zero since the Depot never catches up enough to 
provide any spare spindles to the supply system. 

Figure 19 shows the impact on the readiness of PACFLT's SH-60Bs due solely to 
a nonavailability of main rotor spindles. The plot shows the readiness shortfall (from a 
maximum of 100 percent) caused by the H-60 Depot's delay in providing RFI spindles to 
PACFLT's 84 SH-60Bs flying approximately 100 hours a month. Over a ten year period, 
the Depot delay eventually causes a readiness degradation of approximately ten percent. 

Figures 20 through 22 show the actual model in an object-oriented format. Extend 
modeling software was used to create the SH-60B Logistics Model. The model is divided 
into various sections simulating helicopters (each containing 4 spindles) flying 3.3 hours 
each day at the squadron, spindles being checked and removed at 5000 flight hours, 
spindles being repaired at the Depot, and RFI spindles being passed to supply for transfer 
back to the Fleet. Attributes such as flight hours remaining to overhaul and daily flight 
hours are assigned to each spindle as it passes through the system. Attribute generators 
are assigned constant values or probability distributions with means and standard 
deviations to approximate model variables. The Depot delay, for example, is initiated 
from a random input generator assigned a normal distribution with a mean of 120 hours 
(five days) and a standard deviation of twelve hours. 

The power of this model is that variables such as the length of the Depot delay, 
helicopter flight hours flown per month, the time period of the model, and the number of 
surplus spindles in the supply system can all be changed prior to each run to see the 
impact on the final spindle readiness of PACFLT's helicopters. With a few clicks of a 
mouse on selected objects shown in Figures 20 through 22, any single or multiple 
variable change can be accomplished. The output graphs provide a clear visual display 
of the results of those changes to help strategic managers determine where to best apply 


funding to achieve desired results. 
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Figure 20 Supply Section of the SH-60B Model 
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Figure 21 Flight Operations and Squadron Sections of the SH-60B Model 
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Figure 22 Depot and Squadron Sections of the SH-60B Model 
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The data for the SH-60B Logistics Model were provided by the 
COMNAVAIRPAC's LAMPS MK III Wing. The following assumptions are simulated 


within this model: 


® The LAMPS MKIII Wing "owns" 84 SH-60B helicopters. This model only 
considers these 84 aircraft and does not take into account additional helicopters 
owned by the Naval Reserve, Army, and the Atlantic Fleet. 


e A main rotor spindle must be removed for maintenance every 5000 flight hours 
and sent to the Depot for overhaul. 


® Each SH-60B flies an average of 100 flight hours a month and has four spindles, 
one for each rotor blade. 


@ A spindle has never failed in flight. Spindles are only removed at the mandatory 
5000 flight hour maintenance interval. Cannibalizations or swapping spindles 
between aircraft to optimize readiness was not simulated. A helicopter 1s "down" 
or non-flyable without a RFI spindle installed. 


@ The average turn-around time for a spindle includes three days transit time from 
squadron to Depot, five days repair time at the Depot, three days transit time back 
to the squadron, and three days to reinstall the spindle. 


® It was assumed there are approximately 20 surplus spindles in the supply system. 
@ There are usually zero available RFI spindles in the supply system each month. 
® Once a RFI spindle is received by the squadron, it takes 3 days to install the 
spindle and complete the post-maintenance check flight. 
° To date, a spindle has never been received as non-RF] from the H-60 Depot. 
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